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PREFACE 


This  operational  assessment  was  written  to  document  the  test  and  evaluation  of  the  EG  &,  G 
Training  and  Testing  System  at  the  Los  Angeles  International  Airport  (LAX)  in  1995.  It  was 
tested  according  to  the  Critical  Technical  Issues  and  Criteria  and  Critical  Operational  Issues  and 
Criteria  set  forth  by  the  Federal  Aviation  Administration  (FAA).  These  assessed  the  capability 
of  the  candidate  Screener  Proficiency  Evaluation  and  Reporting  System  (SPEARS).  The  key 
FAA  personnel  who  supported  this  testing  effort  are  J.  L.  Fobes,  Ph.D.,  Aviation  Security 
Human  Factors  (AvSec  HF)  Program  Manager  and  Engineering  Research  Psychologist  for  the 
Aviation  Security  Research  and  Development  Division  (AAR),  D.  Michael  McAnulty,  Ph.D.,  an 
Engineering  Research  Psychologist  with  the  Center  for  Aviation  Simulation  and  Human  Factors. 
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1.  INTRODUCTION. 


1.1  PURPOSE. 


This  Operational  Assessment  (OA)  documents  the  degree  to  which  a  candidate  Screener 
Proficiency  Evaluation  and  Reporting  System  (SPEARS)  device,  the  EG&G  Astrophysics 
Linescan  Training  and  Testing  System  (TnT™),  met  the  functional  requirements  established  by 
the  FAA  as  being  necessary  to  produce  an  effective  workforce  of  airport  X-ray  screeners.  The 
SPEARS  Operational  Test  and  Evaluation  (OT&E)  was  conducted  to  evaluate  the  ability  of  a 
candidate  SPEARS  to  improve,  maintain,  and  monitor  airport  X-ray  screener  performance 
through  the  use  of  Threat  Image  Projection  (TIP)  techniques.  This  OA  addresses  only  the 
evaluation  for  the  on-line  testing  and  training  component.  The  off-line  instructional  training 
portions  of  the  SPEARS  were  investigated  in  an  earlier  T&E  (see  the  Test  and  Evaluation  Report 
[TER]  for  the  SPEARS  Computer-Based  Training  [Fobes,  et  ah,  1995]). 


1.2  SCOPE. 

The  SPEARS  OT&E  was  designed  to  address  the  Critical  Operational  Issues  and  Criteria  (COIC) 
and  the  Critical  Technical  Issues  and  Criteria  (CTIC)  as  set  forth  in  the  SPEARS  Test  & 
Evaluation  Plan  (Fobes  &  McAnulty,  1995).  SPEARS  testing  was  conducted  as  part  of  the 
evaluation  of  the  EG&G  TnT™  on-line  training  and  testing  system.  The  COICs  were  evaluated 
during  operational  testing  at  the  Los  Angeles  International  Airport  (LAX).  Following 
operational  testing,  the  CTICs  were  evaluated  during  technical  testing  in  the  FAA  Aviation 
Security  Laboratory  and  at  the  airport.  The  combined  T&E  focused  on  determining  the  degree 
to  which  the  SPEARS  device  met  the  functional  requirements  embodied  in  both  the  COICs  and 
CTICs,  and  in  complying  with  Human  Factors  Engineering  (HFE)  design  principles. 

1.3  RACKGROUNn. 

1 .3.1  SPEARS  Program  Background. 


The  SPEARS  Program  was  initiated  in  response  to  a  congressional  mandate  (Aviation  Security 
Improvement  Act  of  1990,  Public  Law  101-604)  tasking  the  FAA  to  improve  aviation  security 
through  the  optimization  of  Human  Factors  (HF)  elements  in  the  U.S.  airport  security  system. 
The  issue  of  screener  performance  and  effectiveness  was  highlighted  as  an  area  requiring 
evaluation  to  identify  potential  security  improvements.  An  aviation  security  Department  of 
Transportation  (DOT)  task  force  supported  this  thrust  by  concluding  that  human  performance 
was  the  critical  element  in  the  screening  process. 
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The  mandate  directed  that  screeners  be  effectively  trained  to  use  threat  detection  equipment 
properly.  The  detection  of  explosive  and  incendiary  devices  was  critically  important  because  of 
the  potential  for  significant  damage,  causing  a  crucial  loss  of  life  and  aviation  resources. 


1.3.2  Functional  Description 


The  SPEARS  consists  of  two  components:  (a)  an  off-line  Computer-Based  Training  (CBT) 
system  to  teach  screeners  to  detect  various  threat  objects,  and  (b)  an  on-line  threat  image 
projection  (TIP)  training  and  testing  component  employed  at  airport  security  checkpoints.  This 
latter  configuration  is  to  further  develop  and  maintain  threat  detection  proficiency  by  projecting 
simulated  X-ray  images  of  threat  objects  into  X-ray  images  of  actual  passenger  bags. 

For  the  SPEARS  to  be  effective,  it  must  increase  human  ability  in  the  identification  of  targets, 
namely  threat  objects.  The  aim  of  the  SPEARS  TIP  component  is  to  increase  the  vigilance 
(motivation)  of  screeners  to  detect  threats  within  baggage  through  regular  exposure  to  a  variety 
of  threat  images.  This  exposure  is  accomplished  by  using  TIP  to  insert  Fictional  Threat  Images 
(FTI)  onto  the  X-ray  images  of  normal  baggage  flow  being  X-rayed,  or  to  insert  Combined 
Threat  Images  (CTI)  consisting  of  fictitious  images  of  passenger  bags  which  also  contain  threat 
images  into  the  normal  passenger  bag  image  flow.  TIP  exposure  is  expected  to  enhance 
detection  performance  in  the  following  two  ways. 

a.  Screeners  will  gain  experience  and  thereby  increase  their  ability  to  locate  and 
identify  a  diverse  array  of  threat  objects  under  normal  working  conditions. 

b.  Regular  exposure  to  threat  images  will  increase  screener  attention  levels,  thereby 
maintaining  the  screener’ s  ability  to  detect  threats  in  cluttered  bags. 

Exposure  to  a  large  image  library  of  threat  objects  should  enhance  screener  familiarity  with 
images  of  actual  threat  objects.  Screener  performance  monitoring  and  feedback,  accompanying 
false  image  insertion,  should  also  provide  increased  motivation  and  vigilance.  SPEARS  provides 
training  for  many  categories  of  threats.  Historically,  threat  objects  such  as  weapons  have 
resulted  in  the  highest  detection  rates  in  operational  settings.  In  many  cases,  the  detection  rates 
associated  with  these  types  of  threats  would  be  difficult  to  improve.  Comparatively,  detection 
rates  for  Improvised  Explosive  Devices  (lEDs)  have  been  lower.  lED  detection  performance 
was,  therefore,  of  specific  concern  to  this  study. 


1 .3.3  Literature  Review. 


The  expected  performance  effects  from  the  SPEARS  TIP  component  are  supported  by  a  review 
of  the  applicable  HF  and  psychology  literature,  specifically  in  the  areas  of  Signal  Detection 
Theory  (SDT),  vigilance  theory,  and  training. 
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1.3.3.1  Implications  of  Signal  Detection  Theory  for  the  SPEARS. 


SDT  considers,  in  part,  the  role  of  human  vigilance  in  human  detection  of  infrequent  target 
signals.  A  description  of  SDT  is  provided  in  Appendix  A.  Key  to  the  purpose  of  the  SPEARS 
TIP  is  the  concept  of  vigilance  decrement.  The  vigilance  decrement  has  been  defined  as  a 
decrease  in  human  performance  as  a  function  of  time  on  task  (Wickens,  1992).  In  experimental 
paradigms,  signal  detection  rates  and  signal  detection  measures  (d'  [derived  operator  sensitivity] 
and  P  or  £  [Operator  Response  Criterion])  are  collected  in  time  increments,  with  the  results 
demonstrating  decreasing  signal  detection  performance  over  time.  Although  many  factors  can 
contribute  to  this  decrement,  the  factor  most  pertinent  to  X-ray  screening  of  airline  passenger 
baggage  is  criterion  shifts  associated  with  lower  levels  of  target  expectancy.  Because  the 
frequency  of  lED  presentation  for  the  screener  in  a  real  operational  setting  is  extremely  low,  the 
screener  may  be  unprepared  to  detect  a  threat  object  on  the  rare  occasion  when  one  is  presented. 

In  SDT  terms,  the  expectancy-derived  vigilance  decrement  is  attributed  to  an  upward  adjustment 
of  the  operator  response  criterion  (or  response  threshold)  in  response  to  a  reduction  in  the 
perceived  frequency  (and  therefore  expectancy)  of  target  events  (Wickens,  1992).  If  the  operator 
response  criterion  is  based  on  perceived  target  frequency  levels,  then  additional  missed  targets 
may  result  in  the  perception  of  even  lower  target  frequencies.  The  corresponding  effect  may  be  a 
continual  decrease  in  detection  performance. 

Nadler,  Mengert,  and  Carpenter-Smith  (1994)  indicate  that  inserting  TIPs  into  the  normal  flow  of 
baggage  should  increase  the  expectancy  levels  of  the  baggage  screeners,  thereby  counteracting 
this  adverse  criterion  shift  and  increasing  overall  detection  rates  of  actual  threat  objects.  To 
increase  screener  expectancy  levels,  adequate  and  immediate  knowledge  of  results  for  both  real 
and  artificial  threats  is  required.  TIP  must  also  increase  operator  sensitivity  so  that  the  shift  in 
criterion  is  not  accompanied  by  an  increase  in  false  alarms. 

Operational  screener  watch  durations  are  relatively  short.  For  carry-on  (CO)  baggage  screeners, 
the  watch  duration  ranges  from  20  to  30  minutes,  depending  on  the  security  procedures  in  use  at 
the  particular  airport.  Similarly,  the  watch  durations  for  checked  baggage  screeners  rarely 
exceed  1  hour.  The  majority  of  vigilance  decrement  research  has  focused  on  tasks  with  watch 
durations  of  1  hour  or  greater,  usually  focusing  on  conventional  work  shifts  of  2-,  4-,  or  8-hour 
watches.  While  it  is  doubtful  that  the  degree  of  vigilance  decrements  detected  towards  the  end  of 
the  eighth  watch  hour  are  directly  applicable  to  a  20-minute  or  1  hour  session,  vigilance 
decrement  effects  are  expected  to  accumulate  across  multiple  screening  shifts. 

The  lack  of  actual  threat  presentation  over  several  weeks,  months,  or  years  is  expected  to  result 
in  dramatically  reduced  signal  expectancy.  This  effect  might  be  counteracted  through  the 
systematic  use  of  techniques  (e.g.,  by  regular  insertion  of  FTIs  and  CTIs  into  the  baggage 
screening  environment)  that  change  the  expectancy  of  threat  events.  According  to  SDT, 
therefore,  the  SPEARS  TIP  may  result  in  enhanced  screener  detection  performance  by 
maintaining  appropriate  response  criteria  over  extended  periods. 
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1 . 3.3.2  Implications  of  Training  Theory  on  the  SPEARS. 


The  training  literature  reveals  implications  for  the  design  and  implementation  of  a  SPEARS 
device,  although  the  training  impact  of  TIP  by  itself  should  be  limited.  In  a  review  of  training 
literature,  Goldstein  (1986)  found  that  distributed  practice  for  procedural  skills,  such  as  X-ray 
screening,  provides  the  most  advantageous  results  over  time.  Massed  practice  sessions  tend  to 
show  better  immediate  training  results  and  also  require  less  overall  training  time  to  achieve  a 
minimum  criteria.  Massed,  off-line,  instructional  training  is,  therefore,  a  neeessary  component  of 
screener  training.  For  retention  over  extended  periods,  however,  the  distributed  or  spaced 
training  sessions  will  result  in  better  overall  performance.  TIP,  as  a  regular  component  of  the 
screener’ s  daily  activities,  should  result  in  improved  lED  detection  performance. 

Practice  alone  will  not  provide  adequate  training  retention  in  situations  such  as  the  present  one, 
in  which  the  trainee  cannot  determine  response  accuracy.  Adequate  retention  requires  a  series  of 
interacting  training  variables,  which  includes  an  adequate  and  timely  knowledge  of  results  and 
proper  motivation  of  the  trainees. 

Another  factor  critical  to  the  acquisition  and  subsequent  retention  of  job  skills  is  the  extent  to 
which  training  is  transferred  to  the  operational  environment.  The  TIP  concept  employs  the 
principle  of  identical  elements,  in  that  the  training  task  uses  the  identical  environment  and  stimuli 
to  that  of  the  actual  screening  task.  Training  theory  predicts  that  this  training  principle  should 
result  in  a  high  level  of  transfer  of  TIP  training  to  the  operational  environment,  which  should 
result  in  increased  lED  detection  performance. 


T4_  SYSTEM  DESCRIPTION 


For  this  SPEARS  TIP  OT&E,  only  one  vendor,  EG&G  Astrophysics  TnT™,  agreed  to  have  their 
TIP  system  evaluated. 

1.4.1  EG&G  Astrophysics  TnT™. 


The  following  is  a  technical  description  of  the  TnT^“  components  tested. 

a.  A  color  monitor  and  a  black/white  monitor  mounted  side-by-side,  just  as  the 
monitors  are  mounted  on  the  EG&G  Astrophysics  E-Scan  X-ray  machine. 

b.  A  control  panel  mounted  below  the  monitors  that  replicates  the  control  panel  on 
the  E-Scan  X-ray  machine. 

c.  A  trackball  located  to  the  right  of  the  control  panel  that  operators  use  to  interact 
with  the  training  programs. 

d.  A  computer  located  inside  the  equipment  housing  that  runs  the  training  programs. 
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e.  A  variety  of  cables  and  connectors  used  to  connect  the  training  system  to  the 
EG&G  Linescan  X-ray  and  to  the  recording  equipment,  such  as  video  recorders. 

1.4.2  X-Ray  Machine  Modifications. 


To  allow  for  the  connection  of  TnT^“  workstations,  input  boards  were  installed  by  EG&G 
Astrophysics  to  two  of  the  Linescan  System  Ten  X-ray  machines  at  LAX,  Terminal  2. 

2.  TEST  DESIGN  AND  CONDUCT. 

2.1  EXPERTMENTAT.  DESIGN. 


The  experimental  design  for  the  operational  test  is  an  untreated  control-group  design  with  pretest 
and  posttest  measures  (Cook  and  Campbell,  1979).  This  allowed  for  a  direct  assessment  of  the 
effects  of  the  SPEARS  TIP  intervention  on  screener  lED  detection  performance.  The  basic 
design  was  modified  to  include  a  second  posttest  data  collection  to  measure  the  retention  of  the 
skill  over  time.  Using  Cook  and  Campbell’s  (1979)  notation,  the  experimental  design  is  depicted 
in  Figure  1. 


time—> 


Training  Group 

Ol 

Xi  02  Xi 

03 

Control  Group 

Ol 

02 

03 

FIGURE  1.  EXPERIMENTAL  DESIGN 

Where  X  indicates  the  SPEARS  treatments  (screener  training  on  fictional  images  using  the 
EG&G  TnT™  system),  and  O  indicates  the  ordinal  number  of  screener  performance  tests  where 
01  is  the  pretest,  02  is  the  first  posttest  and  03  is  the  second  posttest.  The  three  tests  had  the 
same  basic  structure  in  that  the  screener  viewed  a  computer-based  series  of  200  X-ray  images  of 
actual  bags  and  had  to  decide  which  ones  contain  a  simulated  lED  and  which  ones  do  not.  Each 
test  used  the  same  bags  but  in  random  order.  No  screener  took  the  same  test  order.  The  large 
number  of  bags  and  the  planned  4  week  time  interval  between  the  tests  should  minimize  the 
chances  of  screeners  memorizing  the  bags. 
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2.1.1  Independent  Variahles. 


There  were  two  independent  variables  other  than  screeners  for  the  SPEARS  TIP  evaluation  in  the 
pretest  and  two  posttests.  The  independent  variable,  SYSTEM,  represents  the  SPEARS  device 
and  has  two  levels:  training  and  control.  The  other  independent  variable,  TRIAL  represented 
the  SPEARS  training  intervention  and  has  three  levels:  pretest,  first  posttest,  and  second 
posttest. 

For  the  TIP  treatments  themselves  (training  group  only),  time  segments  (by  week)  was  the  only 
independent  variable  other  than  screeners.  The  EG&G  system  cannot  record  false  alarms  by  the 
screener  and  therefore  SDT  measures  of  operator  sensitivity  or  response  bias  caimot  be 
computed.  The  data  will  be  divided  into  weekly  segments  for  analysis. 

2.1.2  Dependent  Variables. 


The  dependent  variables  in  the  pretest  and  two  posttests  are  derived  from  the  off-line  screener 
decision  of  lED  presence  for  each  bag  and  were  the  SDT  variables  of  Pd  (probability  of  correct  detection), 
(probability  of  false  alarm  or  incorrect  detection)?  d',  and  £.  The  only  dependent  variable  in  the  analysis  of 
TIP  treatments  was  P  ^  (probability  of  correct  detection).  These  SDT  variables  were  not  available  for  the  TIP 
portion  which  is  restricted  to  only  P^j  for  each  screener  by  each  weekly  segment.  The  calculated 
values  of  Pj,  Pf^,  d',  and  c  were  compared  across  training  conditions. 

For  each  screener,  the  bag  decisions  were  converted  to  hits  and  false  alarms  based  on  the 
correctness  of  screener  yes-no  responses  to  whether  an  lED  was  present.  and  Nfg  are  the  total 
number  of  hits  and  false  alarms  across  all  bags. 

For  each  screener,  was  used  to  calculate  Pj  as  follows: 

Pd  =  Nh/NT 

where  Nj  is  the  total  number  of  test  bag  images  presented. 

Nfa  was  used  to  calculate  Pf^  as  follows: 


Pfa  =  Nfa/Nc 

where  is  the  total  number  of  comparison  bags  presented. 

After  completing  the  lED  test  trials,  the  first  step  in  the  analysis  was  to  separate  the  MBS  bag 
responses  from  the  comparison  bag  responses  and  to  total  these  separately  for  each  subject.  The 
next  step  was  to  convert  the  raw  data  into  a  set  of  hit  and  false  alarm  rates.  The  values  of  Pj  and 
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Pfa  were  then  converted  into  Z),  and  Zfa  by  using  the  normal  tables  and  plotting  against  one 
another 


2. 1.2.1  Predicted  Results. 

Two  hypotheses  were  tested  to  determine  whether  the  SPEARS  is  operationally  effective  for  the 
development  and  maintenance  of  baggage  screening  skills. 

Hypothesis  1  Training:  Screener  performance  in  detecting  lEDs  was  significantly  improved  as  a 
result  of  SPEARS  TIP. 

Hypothesis  2:  Screener  lED  detection  proficiency  was  maintained  over  time  through  the  use  of 
SPEARS  TIP. 

The  first  hypothesis  requires  an  improvement  in  P^  without  a  corresponding  increase  in  Pfa  for 
the  training  group  relative  to  the  control  group.  The  second  hypothesis  requires  that  this 
performance  effect  be  sustained  over  time.  If  this  improvement  in  performance  does  not  occur  or 
is  not  sustained  over  time,  then  the  SPEARS  is  not  operationally  effective  for  either  or  both. 
Some  research  indicates  that  the  performance  should  continue  to  gradually  increase  to  some 
asymptotic  level.  For  this  study,  however,  the  degree  of  this  gradual  increase  is  not  predictable. 
A  hypothetical  representation  of  the  hypothesized  results  is  provided  in  Figure  2. 

Similar  hypotheses  can  be  posited  for  the  effects  of  SPEARS  TIP  on  screener  vigilance,  but  they 
can  only  be  tested  over  time  with  the  training  group.  Therefore,  TIP  performance  will  be  divided 
into  weekly  segments  and  tested  for  the  following  hypotheses. 

Hypotheses  3:  Screener  performance  (Pj)  was  improved  at  detecting  TIPs  during  the  initial 
weeks  of  presentation. 

Hypothesis  4.  Once  Pj  performanee  inereased  to  asymptotic  levels,  it  did  not  deerease  during  the 
later  segments  of  the  study  (see  also  the  trained  performance  line  in  Figure  2). 
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FIGURE  2.  HYPOTHETICAL  OPERATIONAL  TESTING  RESULTS 
2.1.2.2  Statistical  Analysis. 

An  effort  was  made  to  avoid  contamination  of  the  control  group  by  selecting  the  training  and 
control  groups  from  two  separate  terminals  at  LAX.  This  control  strategy  does,  however,  impose 
some  penalty  on  the  statistical  analysis  of  the  resultant  data.  The  primary  penalty  centers  on  the 
assumption  of  equivalent  groups  across  the  two  treatment  groups  (Training  and  Control)  within 
the  System  variable.  Statistical  analysis  procedures  assume  equivalent  groups  or  a  random 
assignment  of  individuals  to  groups.  Without  one  of  these  control  procedures,  any  effects 
discovered  later  in  the  study  could  be  attributed  to  pre-existing  differences  in  the  group  and  not 
to  the  treatment. 

Two  approaches  that  control  for  these  biases  include  experimental  control  and  statistical  control 
(Cook  and  Campbell,  1979;  Kirk,  1982).  In  the  present  study,  the  primary  experimental  control 
was  a  pretest  given  to  both  groups  to  assess  the  magnitude  of  pre-existing  performance 
differences  between  them.  The  pretest  was  very  similar  to  the  two  posttests  that  are  used  to 
assess  the  treatments  effects. 

This  experimental  control  method  was  employed  and  combined  with  a  standard  statistical 
analysis  technique.  A  2-Way  Analysis  of  Variance  (ANOVA)  design  was  implemented  (Kirk, 
1982).  The  resulting  experimental  design  allows  for  analysis  of  the  main  effect  for  System,  the 
main  effect  for  Trial,  and  the  interaction  between  SYSTEM  and  TRIAL  using  screeners’ 
performance  test  scores  as  the  dependent  variable.  Significant  effects  within  System,  Trial,  and 
the  System  x  Trial  interaction  can  be  investigated  using  post  hoc  Neuman-Keuls  tests.  Thus, 
even  if  the  pre-test  revealed  some  preexisting  difference  between  groups,  it  would  be  possible  to 
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meaningfully  interpret  the  training  as  changing  or  having  no  effect  on  the  groups’  inherent 
performance  differences. 

Screener  attrition  resulted  in  unequal  numbers  in  the  training  and  control  groups  at  the  end  of  the 
study.  An  unequal-n  ANOVA  accordingly  was  conducted  since  the  numbers  were  not  overly 
disparate.  There  may  be  some  unavoidable  confounding  effects  due  to  repeated  exposures  of  the 
participants  to  the  data  collection  procedures  used  to  determine  performance.  For  example, 
screener  performance  may  be  improved  over  time  simply  because  the  screeners  are  more 
comfortable  with  the  data  collection  trials,  and  not  due  to  any  training  effect.  However,  this 
should  affect  both  groups  equally. 

These  controls  were  of  less  concern  in  analyzing  the  FTI  performance,  because  all  the  data  were 
collected  on  the  same  group.  A  one-way,  repeated  measures  ANOVA  was  used  to  analyze  the 
effects  of  exposure  to  FTIs  on  FTI  detection. 

2.2  I JSARTT, TTY  ASSESSMENT. 


Descriptive  statistics  were  used  for  the  Likert  Scale  data  collected  on  system  usability.  Any 
Likert  rating  having  an  average  score  of  unacceptable  is  deemed  to  be  a  deficiency. 
Recommendations  were  made  to  rectify  the  deficiencies.  Any  software,  hardware,  or  procedural 
aspects  that  were  marked  as  failures  on  the  HFE  checklist,  received  negative  comment  by  the 
screeners,  or  were  noted  during  operation  of  the  SPEARS  equipment  are  reported  and 
recommendations  for  improvement  made. 


2.3  OPERATIONAL  TEST  PROCEDURES. 


2.3.1  Subjects. 


Two  groups  of  subjects,  a  training  group  and  a  control  group,  were  used  to  conduct  the 
operational  assessment  of  the  SPEARS  on-line  training  and  testing.  The  control  group  did  not 
receive  SPEARS  TIP  training  during  the  OT&E.  To  the  extent  that  the  control  group  was  aware 
of  the  activities  of  the  training  group,  the  members  might  have  exhibited  decreased  performance 
over  time  caused  by  not  being  selected  for  special  treatment.  To  reduce  this  effect,  the  training 
and  control  groups  were  physically  separated  by  conducting  testing  at  LAX  in  two  separate 
terminals. 

The  initial  sample  consisted  of  30  baggage  screeners  selected  from  the  day  (0630-1500  hours) 
and  swing  (1430-2300  hours)  shifts  at  LAX  Terminal  7  (United  Airlines)  and  24  screeners 
selected  from  the  day  and  swing  shifts  at  LAX  Terminal  2  (Northwest  Airlines).  The  number  of 
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subjects  for  each  group  was  chosen  to  account  for  anticipated  attrition.  (The  attrition  rates 
differed  in  the  two  terminals.)  The  Terminal  7  screeners  participated  in  the  control  group  and 
the  Terminal  2  screeners  participated  in  the  training  group. 

2.3.2  Subject  Training. 


The  screeners  were  briefed  about  the  study  in  terms  of  their  activities,  schedules,  and 
expectations  as  participants.  They  also  completed  a  consent  form  and  a  personal  data  form  (see 
Appendices  L  and  M)  before  participating  in  the  study.  The  screeners  received  protocol 
instructions  (see  Appendix  N)  before  the  study  and  immediately  before  each  major  test  activity. 

Because  training  and  experience  varied  across  subjects,  all  participants  attended  a  lecture  session 
of  X-ray  screening  philosophy  and  procedures,  image  interpretation,  and  possible  threat  objects. 
This  training  sought  to  equalize  knowledge  across  subjects.  The  training  required  approximately 
2  hours  to  complete. 

Training  on  the  use  of  the  specific  SPEARS  device  was  provided  to  the  entire  Terminal  2 
screening  staff.  Because  of  the  flexibility  inherent  in  airport  baggage  screener  duty  cycles,  there 
was  a  chance  that  a  Terminal  2  screener  might  inadvertently  receive  a  TIP,  even  though  the 
screener  was  not  a  member  of  the  training  group.  To  account  for  this  eventuality,  all  Terminal  2 
screeners  received  training  to  recognize  and  properly  resolve  a  TIP. 


2.3.3  Pilot  Study. 


A  pilot  study  was  condueted  approximately  2  weeks  before  the  OT&E  to  verify  the  test  protocol 
and  data  collection  procedures.  Data  collection,  the  SPEARS  TIP,  and  lED  detection  testing 
procedures  were  verified.  Two  screeners  participated  in  the  pilot  study,  one  from  each  of  the 
treatment  groups. 

2.3.4  Operational  Test  Protocol 


The  protocol  describes  the  manner  in  which  data  were  gathered  to  assess  the  operational 
effectiveness  of  SPEARS  TIP .  The  specific  protocol  for  each  of  these  data  collection  exercises  is 
described  in  sections  2.3.4. 1  through  2. 3.4. 4. 

Screening  activities  at  LAX  were  carried  out  24  hours  per  day,  and  were  divided  into  three  shifts. 
To  obtain  the  required  screener  sample  size,  test  events  were  carried  out  during  the  morning  shift 
(0630-1500  hours)  and  the  swing  shift(1430-2300).  The  two  shifts  overlap  for  30  minutes 
between  1430-1500  hours  to  ensure  a  smooth  transition  between  shifts.  During  the  transition,  the 
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swing  shift  operates  the  screening  stations  and  is  monitored  by  the  morning  shift.  Because  data 
must  be  collected  during  two  shifts,  there  were  two  equivalent  data  collection  cycles  during  each 
day. 

All  briefing  and  data  collection  activities  took  place  in  Room  2034  and  at  the  Security 
Checkpoint  of  Terminal  2  and  in  the  Aloha  Room  of  Terminal  7.  Diagrams  of  the  terminals  and 
the  rooms  are  provided  in  Appendix  O. 

2. 3.4.1  Pretest. 


The  pretest  was  conducted  to  assess  the  baseline  lED  detection  performance  of  the  screeners 
participating  in  the  study.  Schedules  for  the  pretest  are  included  in  Appendix  P.  Both  the 
training  and  control  groups  were  pretested  concurrently.  The  geographic  separation  between  the 
training  and  control  groups  required  two  sets  of  test  administrators,  one  per  group. 

Data  were  collected  on  days  one  through  three  of  the  pretest  and  each  screener  was  required  once 
for  a  2-hour  period.  Participants  were  also  administered  the  Snellen  equivalent  High  (96  percent) 
Contrast  Vision  Acuity  Test  (see  Appendix  Q  for  the  score  sheet). 

Following  selection  as  a  study  participant,  each  screener  was  assigned  a  subject  number  which 
was  used  to  anonymously  identify  the  screener  during  the  study.  Participants  then  received  the 
computerized  lED  detection  performance  test.  Before  beginning  the  performance  measurement, 
the  data  collector  briefed  each  screener  using  the  instructions  contained  in  Appendices  N,  R,  S, 
and  T.  Each  screener  was  informed  that  simulated  explosive  devices  would  be  present  in  some 
of  the  bag  images  to  be  screened  during  the  test  trial.  Screeners  carried  out  normal  screening 
operations  and  did  not  receive  operational  direction  from  the  test  administrators. 

This  test  required  approximately  35  minutes.  Before  each  test  session,  the  FAA  data  collector 
assigned  an  operator  identification  number  to  each  screener  for  access  to  the  test  device  and 
ensured  that  the  test  device  was  operable  and  properly  configured.  If  the  test  device 
malfunctioned  during  the  test  trial,  the  data  collector  informed  the  test  manager,  who  arranged 
for  maintenance. 

The  data  collector  started  the  test  trial  and  asked  the  screener  to  begin  the  test.  The  testing  device 
automatically  displayed  the  first  bag  image  to  the  screener.  The  screener’s  task  was  to  indicate 
whether  the  bag  contained  a  threat  image,  within  1 0  seconds,  by  pressing  the  appropriate  key  on 
the  keyboard.  After  the  screeners  had  responded  to  this  initial  question,  they  were  asked  a 
second  question  based  upon  their  response  to  the  first  question.  They  were  asked  to  indicate  how 
confident  they  were  in  their  original  response  concerning  the  presence  or  absence  of  an  lED  in 
the  bag.  The  testing  device  recorded  the  response  and  automatically  forwarded  to  the  next 
image.  The  data  collector  stopped  the  test  when  all  bag  images  had  been  presented. 
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Each  computerized  test  consisted  of  25  test  bag  images  containing  Modular  Bomb  Sets  (MBS) 
within  a  series  of  175  CO  comparison  bag  images  for  a  total  of  200  bag  images.  A  description  of 
the  test  bag  configuration  is  included  in  Appendices  U  and  V.  All  screeners  in  the  study  viewed 
the  same  test  and  comparison  bag  images.  To  control  for  presentation  order,  six  randomly 
generated  presentation  orders  for  the  200  bag  images  were  used. 

The  control  group  returned  to  normal  duties  after  the  lED  test.  Instruction  in  the  operation  of  the 
SPEARS  TIP  training  equipment  and  the  procedure  for  threat  resolution  when  the  TnT™  TIP 
feature  becomes  operational  was  provided  to  the  Terminal  2  staff.  The  participants  were  then 
returned  to  their  normal  duties. 


2.3.4.2  On-line  TIP  Testing 


The  basic  task  of  the  X-ray  screener  is  to  monitor  X-ray  images  presented  on  display  screens  and 
identify  potential  threat  objects  within  the  images.  The  task  for  both  SPEARS  TIP  and  lED 
detection  testing  matched  this  basic  operational  task:  Screeners  examined  bags  and  were  to 
identify  those  that  contained  a  potential  threat  object. 

The  SPEARS  TIP  task  involved  the  X-ray  screeners  carrying  out  normal  screening  duties  at  their 
usual  screening  station  (Terminal  2  checkpoint).  Tn’P'^  devices  were  connected  to  two  of  the 
three  X-ray  machines  at  the  Terminal  2  checkpoint,  as  depicted  in  Appendix  O.  The  TnT™ 
devices  were  configured  to  have  the  TIP  function  insert  up  to  24  FTIs  into  the  normal  passenger 
baggage  flow  to  be  observed  by  each  screener  on  a  given  day. 

The  method  for  FTI  selection  and  insertion  was  as  follows.  The  number  of  FTIs  presented  to 
each  screener  on  a  given  day  is  detailed  in  the  randomly  generated  protocol  shown  in  Appendix 
W.  During  each  8-hour  shift,  the  duty  cycle  for  a  screener  normally  consisted  of  a  20-minute 
period  on  the  X-ray  equipment,  20  minutes  using  the  wand  for  body  searches,  20  minutes 
performing  bag  searches,  and  20  minutes  off.  A  normal  shift  also  included  a  30-minute  lunch 
break.  This  duty  cycle  results  in  each  screener  being  on  the  X-ray  equipment  a  maximum  of 
once  per  hour,  and  a  total  of  approximately  four  sessions  per  shift.  Because  of  the  inherent 
flexibility  of  screening  operations,  it  is  impossible  to  determine  when  these  four  sessions  will 
occur  for  each  screener.  For  this  reason,  each  hour  of  each  shift  was  divided  into  three,  20- 
minute  segments.  The  number  of  FTIs  presented  to  a  screener  on  a  given  day  was  doubled  and 
spread  across  the  eight-hour  shift.  The  number  of  FTIs  presented  in  each  hour  was  replicated  in 
each  20-minute  segment.  Using  this  procedure,  reasonable  assurance  was  provided  that  each 
screener  would  be  able  to  receive  close  to  the  required  number  of  FTIs  and  that  all  screeners 
could  receive  approximately  the  same  FTI  exposure  on  a  given  day.  The  flight  schedule  for 
Terminal  2  is  included  in  Appendix  X.  The  ebb  and  flow  of  passengers  resulted  in  uneven  FTI 
presentation  rates  across  screeners.  This  is  unavoidable  given  the  operational  environment  and 
the  way  the  device  can  be  programmed. 
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When  the  X-ray  screener  believed  a  threat  object  was  present  in  a  bag,  the  screener  pressed  the 
TnT™  TIP  button  to  elicit  a  response  from  the  SPEARS  device.  If  the  bag  contained  an  FTI,  the 
SPEARS  device  confirmed  the  fictional  threat  object  and  indicated  it  on  the  display.  The  threat 
object  was  then  erased  from  the  screen,  allowing  the  X-ray  screener  to  ensure  that  there  were  no 
real  threat  objects  in  the  bag.  Only  one  FTI  per  bag  was  generated  by  the  SPEARS  device,  but 
there  was  a  possibility  of  an  actual  object  in  the  bag.  If  the  bag  did  not  contain  an  FTI,  but  the 
screener  indicated  a  possible  threat  through  the  TIP  key,  notification  of  no  projection  was  given 
to  the  screener,  and  normal  security  procedures  were  followed  (see  Appendix  T.  If  the  screener 
failed  to  detect  the  FTI,  the  TnT™  notified  the  screener  of  the  miss  and  recorded  the 
presentation  in  the  screener’ s  database. 

2. 3.4. 3  First  Posttest. 


The  first  posttest  was  conducted  6  weeks  after  the  pretest  and  consisted  of  lED  detection  testing 
only.  The  control  group  had  24  screeners  remaining  (out  of  30)  and  the  training  group  had  23 
screeners  (out  of  24).  This  testing  followed  the  same  protocol  contained  in  the  pretest 
description.  Each  screener  was  required  to  take  a  test  session  lasting  approximately  35  minutes. 
The  first  posttest  required  2  days  to  complete.  The  control  group  and  training  group  were  tested 
concurrently. 

2. 3.4.4  Second  Postte.st. 


The  second  posttest  was  conducted  8  weeks  after  the  first  posttest  and  again  involved  a  35- 
minute  lED  detection  test  session  for  each  screener.  The  control  group  had  20  screeners 
remaining  (of  30)  and  the  training  group  had  17  screeners  remaining  (of  24).  The  second  posttest 
had  to  be  delayed  by  several  weeks  due  to  heightened  security  alerts  during  this  time  period  at 
LAX.  Following  lED  detection  testing,  each  screener  in  the  training  group  completed  a  usability 
assessment  form.  The  second  posttest  required  3  days  to  complete.  The  training  group  and 
control  group  were  tested  concurrently. 


2.3.5  SPEARS  Usability  Assessment  Methodology. 


The  tailored  Usability  checklist  was  completed  during  the  testing  of  the  technical  issues.  The 
subjective  ratings  covered  the  SPEARS  operating  procedures  and  feedback  provided  by  the 
SPEARS  device  during  real  time  baggage  screening.  The  ratings  were  provided  through  5-point 
Likert  scales  (see  Appendix  C)  developed  for  the  SPEARS  training  and  testing  and  administered 
to  each  screener  in  the  training  group  during  the  second  posttest  session.  The  structured 
assessment(see  Appendix  B)  was  conducted  to  probe  the  usability  of  the  SPEARS  device  and  are 
complementary  to  the  Likert  scale  ratings. 
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2.3.6  Technical  Test  Procedure.s. 


Technical  test  procedures  are  described  in  the  Test  Strategy  Results  for  each  Technical  Issue. 
Most  technical  test  data  were  collected  by  Human  Factors  Engineers  using  observation  and 
tryouts  based  on  checklists  appropriate  for  each  domain.  The  checklists  are  presented  in 
Appendices  D  through  K. 


14  TEST  LIMITATIONS  AND  IMPACT 


A  number  of  known  or  unforeseen  constraints  and  events  during  the  conduct  of  this  test  may 
have  had  some  influence  on  the  results  or  their  interpretability. 

Screeners  reported  difficulty  using  the  trackball.  In  a  few  instances,  this  may  have  affected 
responding  in  a  timely  manner  to  the  images. 

Screeners  often  used  other  operators’  IDs  to  login,  possibly  to  prevent  their  results  from  being 
identified.  In  about  5%  -  10%  of  these  cases,  it  was  not  possible  to  reconstruct  which  individual 
had  taken  a  particular  session.  This  had  an  approximately  random  pattern  of  occurrence. 

It  was  not  possible  to  ensure  that  all  screeners  in  the  training  group  got  a  uniform  amount  of 
exposure  to  the  TIP  treatment.  Some  subjects  had  no  TIP  practice  for  several  weeks.  Others  had 
consistent  but  extremely  limited  amounts  of  exposure  to  the  items,  e.g  2-4  items  for  several 
weeks  in  a  row.  Thus,  it  is  not  clear  what  training  benefit  might  be  expected  for  those  screeners 
who  received  limited  or  intermittent  TIP  exposure. 

As  noted,  the  training  aspects  of  detecting  the  FTIs  were  examined  using  a  computer-based 
device  to  present  the  black  and  white  X-ray  image.  The  primary  reason  for  using  the 
computerized  device  rather  than  actual  baggage  containing  MBSs  was  to  avoid  the  negative 
effects  that  multiple  trials  using  actual  bags  would  have  had  on  passenger  flow  and  airline 
operations.  The  computerized  presentation  of  images  lacks  some  aspects  of  operational 
representativeness  but,  on  the  other  hand,  improved  the  probability  of  retaining  the  screener’s 
full  attention  to  the  task.  The  screeners  analyzed  actual  X-ray  images  and  had  to  respond 
quickly  to  each  image,  simulating  the  passenger  flow  requirements  at  the  checkpoint. 

The  SPEARS  TIP  is  meant  to  be  tailored  by  supervisors  to  individual  screener  abilities,  training 
needs,  and  workload.  Such  tailored  training  and  testing  could  not  be  done  during  this  OT&E, 
and  a  more  generic  training  and  testing  protocol  was  used  (see  section  2.4.2. 1).  Therefore,  the 

SPEARS  TIP  training  could  not  be  implemented  in  optimal  fashion  for  all  of  the  screeners  in  this 
study. 
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3.  RFSIJTTS  AND  ANAT.YSTS:  OPERATIONAL  AND  TECHNICAL  TEST, 


This  section  presents  the  data  and  associated  analysis  for  each  Critical  Operational  Issue  and 
Critical  Technical  Issue  as  defined  by  its  Criteria,  Measures  of  Performance  (MOPs),  and  test 
strategy. 

The  issues  were  investigated  through  screener  performance  data  collected  during  the  study,  and 
through  subjective  comments  and  ratings  collected  after  the  study’s  completion.  Additional  data 
for  the  usability  issue  were  collected  by  using  a  tailored  Usability  HFE  Checklist  (Appendix  B) 
based  on  the  Guidelines  for  the  Design  of  User  Interface  Software  (Smith  and  Mosier,  1986). 


SPEARS  TIP  Training  and  Testing  Data. 

For  each  of  the  pretest  and  the  two  posttests,  the  screener  decision  and  confidence  responses 
observed  while  scanning  for  simulated  lEDs  in  the  computerized  passenger  bag  X-ray  images 
were  collected  for  each  screener  in  both  groups.  During  the  TIP  training  periods,  data  were 
collected  on  the  number  of  TIP  hits  and  misses  for  each  screener. 

SPEARS  Usability  Data. 

The  data  resulting  from  the  SPEARS  usability  analysis  were  the  HFE  checklist  data,  the  Likert 
scale  responses,  and  any  procedural  errors  noted  when  the  training  group  subjects  operated  the 
SPEARS  device. 

3.1  OPERATIONAL  CRITERIA  AND  RESULTS. 


3.1.1  Operational  Issue  1 :  Training  Effectiveness 


Does  TIP  enhance  screener  performance? 

Criterion  A.  Projecting  fictional  images  is  accompanied  by  a  sustained  increase  (statistical)  in 
the  screener’ s  probability  of  detection  (P<j)  of  lEDs. 

Criterion  B.  Projecting  fictional  images  is  not  accompanied  by  a  sustained  increase  (statistical) 
in  the  screener’ s  probability  of  false  alarm  (PfJ  when  scanning  for  lEDs. 

Criterion  C.  Projecting  fictional  images  is  accompanied  by  a  sustained  increase  (statistical)  in 
the  screener’ s  d'. 
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3.1. 1.1  Test  Strategy. 


MOP  data  were  collected  concurrently  for  two  groups  of  screeners  during  the  operational  testing 
at  LAX.  One  group  of  screeners,  the  experimental  group,  was  exposed  to  the  SPEARS  TIP.  The 
other,  the  control  group,  was  tested  without  exposure  to  the  SPEARS  TIP.  To  determine  the 
effect  of  SPEARS  training  on  lED  detection  performance,  Nh  and  Nfa  was  recorded  for  individual 
screeners  three  times  during  the  study:  (a)  during  a  pretest  before  the  SPEARS  TIP  training 
begins  for  the  experimental  group,  (b)  during  a  posttest  6  weeks  after  the  pretest,  and  (c)  during  a 
second  posttest  8  weeks  after  the  first  posttest. 

The  MOP  data  were  used  to  derive  Pj  and  Pfa  for  each  screener.  The  values  of  P^  and  Pfg  were 
then  compared  between  training  and  control  conditions  and  across  measurement  periods  to 
determine  the  effect  of  the  SPEARS  TIP  on  lED  detection  performance. 


3.1 .1 .2  MOPS  and  Data  Presentation. 


MOP  1-  The  Number  of  Hits  (Nh)  for  test  bag  images  collected  before  and  after  exposure 
to  the  SPEARS  treatment. 

NOTE:  The  number  of  hits  for  training  and  control  group  subjects  is  included  as  part  of 
the  analysis  of  d’  in  MOP  3  of  this  section. 


MOP  2.  The  Number  of  False  Alarms  (Nfa)  for  comparison  bag  images  collected  before 
and  after  exposure  to  the  SPEARS  treatment. 

NOTE:  The  number  of  false  alarms  for  training  and  control  group  subjects  is  included  as 
part  of  the  analysis  of  d’  in  MOP  3  of  this  section. 

MOP  3.  The  d'  to  detect  the  test  bag  images  collected  before  and  after  exposure  to  the 
SPEARS  treatment. 

The  operator  sensitivity  measure  (d’)  means  are  displayed  in  Figure  3  for  the  training  and  control 
groups  for  the  pretest,  posttest  1  and  posttest  2.  Test  completion  times  and  the  signal  detection 
performance  of  the  two  groups  were  compared  across  test  trials.  Five  two-way  ANOVAs  with 
one  repeated  measure  were  conducted  with  group  (training,  control)  and  trial  (pretest,  posttest  1, 
posttest  2)  as  the  independent  variables.  Trial  was  also  a  repeated  measure  since  the  same 
subjects  took  all  three  tests.  The  dependent  variables  were  test  time,  Pd,  Pfa,  d’  or  p. 
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The  ANOVAs  revealed  significant  differences  between  groups  on  screener  sensitivity  but  not  in 
the  expected  direction  nor  differentially  across  trials.  That  is,  the  control  group  had  higher  mean 
screener  sensitivity  (d’)  on  the  pretest  and  both  posttests  than  did  the  training  group  (F  2,  70)  = 
9.44,  p  <  .01).  However,  there  was  no  significant  interaction  between  trial  and  group.  That  is, 
the  difference  between  groups  remained  stable  across  trials.  Both  groups  also  had  statistically 
equivalent  test  completion  times.  There  were  no  significant  differences  between  groups  or  across 
trials  with  Pd,  Pfa,  or  p. 

Mean  screener  sensitivity,  d’,  increased  significantly  across  trials  for  both  groups  (F  1,  35  = 
4.21,  p  <  .05).  The  training  group  had  a  significant  increase  in  sensitivity  going  from  a  mean 
sensitivity  of  .79  at  the  pretest  to  1.28  at  posttest  2  (Duncan  post-hoc,  p<  .05).  The  control  group 
had  a  significant  increase  from  1.14  in  the  pretest  in  1.44  in  the  first  and  second  posttests 
respectively.  (Duncan  post  hoc,  p<.05) 


Fig.  3  MEAN  OPERATOR  SENSITIVITY  ACROSS  TRIALS 


3. 1.1.3  Analysis  and  Discussion. 


The  ANOVAs  indicate  that  the  screener  performance  was  little  influenced,  if  at  all,  by  the  TIP 
training.  The  higher  sensitivity  of  the  control  group  on  the  pretest  compared  with  the  training 
group  was  unexpected,  but  in  no  way  compromises  the  analysis.  The  TIP  treatments  simply  need 
to  increase  training  group  posttests  performance  at  a  faster  rate  than  occurs  from  the  control 
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group  in  order  to  demonstrate  their  effectiveness.  However,  a  significant  increase  in  sensitivity 
was  found  in  both  the  training  group  and  the  control  group  across  trials.  Since  there  was  no 
significant  increase  in  hit  rate  or  false  alarm  rate,  it  is  unclear  how  much  weight  to  accord  this 
isolated  result. 

An  examination  of  the  performance  on  individual  test  items  (on  the  pretest)  did  not  reveal  any 
items  which  were  either  overly  easy  or  overly  hard,  which  might  have  skewed  the  results.  Only 
two  items  were  gotten  right  by  more  than  85%  of  the  54  pretest  takers.  No  item  was  missed  by 
everyone. 

The  reason  that  mean  screener  sensitivity  increased  for  both  groups  cannot  be  determined  on  the 
basis  of  the  data.  However,  typical  causes  for  such  general  non-treatment  based  increases  in 
performance  include  greater  familiarity  with  testing  procedures  or  the  kinds  of  items  presented, 
less  stress  on  the  second  and  third  sessions  and  attrition  of  poorer  performers.  Perhaps  most 
importantly,  security  levels  were  raised  at  the  airport  after  the  test  had  begun.  One  or  more  of 
these  factors  probably  account  for  the  observed  findings. 


3.1.2  Operational  Issue  2:  Vigilance. 

Does  SPEARS  TIP  increase  and  sustain  screener  vigilance  in  detecting  threat  images? 

Criterion  A.  There  is  a  sustained  increase  (statistical)  in  Pj  of  FTIs  for  the  TIP  training  group 
during  the  first  4  weeks  of  the  study. 

Criterion  B.  There  is  a  sustained  increase  (statistical)  in  Pj  of  FTIs  for  the  TIP  training  group 
during  the  last  8  weeks  of  the  study. 


3. 1.2.1  Test  Strategy. 


MOP  data  were  collected  during  OT  at  LAX.  The  SPEARS  device  was  installed  at  the  airport 
and  used  by  the  experimental  group  of  screeners  over  an  extended  period.  To  determine  the 
effect  of  SPEARS  TIP  on  screener  vigilance,  the  number  of  FTI  presentations  and  Nh  were 
observed  and  recorded  for  the  experimental  group  screeners  throughout  the  SPEARS 
intervention.  Unfortunately,  the  SPEARS  device  does  not  record  false  alarms  or  the  total  number 
of  passenger  bags  screened,  so  other  measures  (e.g.,  SDT  measures)  cannot  be  computed.  The 
resulting  Pd  measures  were  compared  across  weekly  segments  of  the  study  to  determine  if 
screener  performance  increases  from  initial  levels  and  is  then  maintained  over  time. 

3.1 .2.2  MOPS  and  Data  Presentation. 


M.QP  5.  The  N|,  for  FTIs  collected  across  the  first  6  weeks  of  the  FTI  treatment. 
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NOTE:  Due  to  the  delay  in  testing  caused  by  enhanced  security  operations  at  LAX,  the 
first  part  of  the  test  was  extended  by  2  weeks  beyond  the  original  schedule  of  4  weeks. 

The  mean  hit  rate  across  weeks  of  TIP  presentation  are  shown  in  Figure  4.  There  was  a  large 
jump  in  hit  rate  between  weeks  2  and  3  when  the  performance  went  from  .36  to  .51  hit  rate. 
After  this  one  time  jump,  the  training  groups’  performance  remained  quite  consistently  at  this 
level..  A  series  of  within-subjects  ANOVAs  of  FTI  Pd  by  week  was  performed.  Consistent  with 
the  differences  noted  above,  a  significant  effect  was  observed  for  the  weeks  before  the  first 
posttest  (F=3. 55,  p<.01) 
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MOP  6.  The  N|,  for  FTIs  collected  across  the  last  8  weeks  of  the  FTI  treatment. 

The  hit  rate  remained  at  the  levels  seen  after  week  2  in  the  first  part  of  the  test.  There  was  a  jiggle 
in  weeks  8  and  9  as  hit  rate  went  down  and  then  back  up  to  the  levels  seen  in  the  first  half  of  the 
test.  This  was  the  basis  for  the  significant  effect  on  screener  Pd  for  the  weeks  between  the  first 
and  second  posttests  (F=2.92,  p.  <.05).  Only  one  screener  out  of  18  showed  a  significant 
increase  in  performance  over  the  14  week  period  however,  (Duncan  post-hoc,  p.  <  .05).  No 
significant  effects  were  found  between  screener  FTI  Pd  between  the  first  and  second  posttests. 
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Analysis  and  Discussinn 


A  treatment  effect  on  vigilance  would  have  some  of  the  characteristics  that  are  seen  in  the  data 
here.  That  is,  if  TIP  presentations  led  to  an  increase  in  attending  to  and  analyzing  X-ray  images, 
then  we  should  see  a  relatively  rapid  rise  in  performance  that  is  sustained  over  the  period  of  the 
treatment.  However,  an  increase  in  vigilance  can  occur  for  a  variety  of  reasons  and  not  just  as 
the  result  of  TIP.  During  the  time  period  in  question,  LAX  had  several  increases  in  security  level 
due  to  concerns  about  the  Trade  Center  trial  and  threats  made  by  the  UnaBomber.  This  may  well 
have  heightened  the  screeners’  focus  on  detecting  threats  in  baggage.  The  significant  increase  in 
control  group  sensitivity  supports  this  interpretation. 

As  noted  in  the  section  on  Test  Limitations,  the  TIP  treatment  could  not  be  operationally 
administered  in  uniform  fashion  to  all  members  of  the  training  group.  This  was  due  to  a  serious 
system  design  limitation  which  precluded  the  linkage  of  TIP  presentations  with  individual  ID 
numbers.  Since  the  screeners  do  not  follow  strict  schedules  at  each  station,  there  was  no  way  to 
know  when  an  individual  was  going  to  be  engaged  in  X-ray  screening.  Thus,  the  amount  of 
training  and  vigilance  benefit  was  compromised  to  some  extent.  The  variability  of  TIP  exposure 
was  quite  large.  One  screener  received  130  TIP  exposures  over  14  weeks.  At  the  other  end,  one 
screener  had  26  exposures  over  the  14  week  period  and  had  no  exposure  on  5  of  those  weeks. 
The  analysis  of  individual  screener  trend  data  over  trials  revealed  only  one  screener  in  the 
training  group  whose  performance  increased  significantly  over  the  course  of  the  TIP  treatments 
themselves  (the  subject  received  79  TIP  exposures,  incidentally). 

The  essential  problem  with  the  TIP  as  presented  by  the  system  is  that  the  lack  of  appropriate 
feedback  and  inconsistency  of  presentation  schedule.  These  make  it  highly  unlikely  that 
screeners  can  improve  their  skill  at  detecting  threats.  If  screeners  don’t  already  have  all  the 
necessary  training  and  demonstrated  ability  to  detect  these  threat  images  prior  to  TIP  then  they 
will  not  acquire  them  from  TIP  which  lacks  such  capabilities.  Thus,  it  is  probable  that  the  false 
alarm  rate  went  up  as  well  when  the  hit  rate  went  up,  although  this  could  not  be  determined  due 
to  limitations  of  the  report  capabilities  of  the  TnT™. 


3.1.3  Operational  Issue  3:Usabilitv. 


Are  there  any  software  or  hardware  factors  or  procedural  aspects  that  degrade  the  screeners’  or 
supervisors’  ability  to  use  the  system  effectively? 

Criterion  A.  Investigative  in  nature. 
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3.1.3.1  Test  Strategy. 


The  usability  issue  is  relevant  to  the  supervisors  who  program  the  machine  and  to  the  screeners 
who  must  detect  and  respond  to  the  TIP  capability  (i.e.,  TIP  button  on  the  TnT^^).  The  usability 
evaluations  consisted  of  completion  of  rating  scales  by  the  participating  screeners  and 
supervisors  (Appendix  C).  Data  also  included  the  application  of  a  tailored  SPEARS  Usability 
Human  Factors  Engineering  (HFE)  checklist  (see  Appendix  B)  based  on  the  Guidelines  for  the 
Design  of  User  Interface  Software  (Smith  and  Mosier,  1986)  and  MIL-STD-1472D. 


3.1.3.2  MOPS  and  Data  Presentation 


MOP  7.  Subjective  ratings  of  system  usability  by  screeners  and  supervisors. 

The  mean  screener  responses  items  in  the  Usability  Survey  are  presented  in  Table  1,  (See 
Appendix  C  SPEARS  OT&E  Screener  Usability  Ratings,  for  questions).  All  but  two  mean  item 
responses  had  a  narrow  range  from  3.35  to  3.93  (5  point  scale:  1  -  minimum,  5  -  maximum) 
indicating  neutral  or  slightly  positive  opinions  concerning  the  utility  and  ease  of  use  of  the  TIP 
training  and  testing.  The  two  highest  rated  items  were  items  5  (4.65)  and  6  (4.12)  which  dealt 
with  using  the  TIP  function  and  operating  the  TIP  key  on  the  computer.  Screeners  felt 
moderately  positive  concerning  the  key  desired  characteristics  of  the  Tn'D’^  system,  namely,  its 
efficacy  in  improving  the  ability  to  detect  explosives  £ind  its  desirability  as  an  on  the  job  training 
tool. 


TABLE  1 .  MEAN  SCREENER  USABILITY  RATINGS 


Question 

Valid  N 

Mean 

Std.  Dev. 

01 

17 

3.82 

1.19 

02 

3.76 

1.35 

03 

3.59 

1.66 

04 

3.24 

1.72 

05 

4.65 

0.70 

06 

4.12 

1.22 

07 

16 

3.75 

1.61 

08 

17 

3.88 

1.17 

09 

15 

3.60 

1.40 

16 

3.75 

1.48 

11 

17 

3.65 

1.50 

12 

17 

3.35 

1.41 

21 


13 

14 

15 

17 

17 

17 

3.35 

3.93 

3.82 

1.73 

1.49 

1.59 

MOP  8.  HFE  Checklist  results. 


The  TnT™  has  mostly  acceptable  usability  properties,  although  some  significant  deficiencies 
were  noted  which  should  be  corrected.  The  most  troublesome  deficiencies  lay  in  the  domain  of 
user  guidance.  The  system  did  not  display  error  messages  or  offer  other  guidance  to  the  operator 
concerning  invalid  entries.  It  also  provided  no  status  information  about  on-going  operations, 
system  delays  or  other  activities,  e.  g.,  no  confirmation  of  ID  number.  The  Help  feature  was  not 
working  either.  A  variety  of  other  user  interface  deficiencies  were  noted.  The  screen  had  low 
contrast  and  was  difficult  to  read  in  the  relatively  dim  airport  environment.  No  highlighting  or 
other  forms  of  enhancement  were  used  to  make  important  or  changing  information  stand  out. 
Coloration  in  graphs  was  arbitrary  and  meaningless.  There  was  no  Undo  command  to  correct  an 
immediately  preceding  error.  Users  could  not  reverse  their  path  on  some  menus  to  return  to  the 
preceding  (higher )  menu.  In  addition,  cut/paste  editing  functions  were  not  available. 

3.1 .3.3  Analysis  and  Di.scussion. 


This  issue  is  judged  Partially  Met,  since  some  interface  deficiencies  exist  which  absolutely 
require  correction.  It  also  should  be  noted  that  screeners  had  only  mildly  positive  opinions  about 
the  system’s  effectiveness  in  promoting  skills  in  detecting  lEDs.  The  lack  of  feedback  on 
responses  and  absence  of  status  messages  is  the  most  serious  interface  deficiency.  This  may  have 
contributed  to  operators’  lukewarm  opinion  of  its  utility  since  they  could  not  really  learn  what 
characteristics  of  threats  were  causing  them  the  most  problem  and  try  to  learn  how  to  better 
recognize  them.  If  the  TIP  is  to  be  used  purely  as  a  motivator,  then  screeners  must  be  fully 
trained  on  the  detection  of  threats  prior  to  exposure  to  TIP. 


3.2  TECHNICAL  CRITERIA  AND  P  RST  TT  .TS 


This  section  identifies  each  Critical  Technical  Issue,  its  related  criteria,  MOPs,  and  test  strategy 
to  be  addressed  by  OT&E.  Several  of  the  technical  issues  for  this  OT&E  are  redundant  with 
technical  issues  tested  in  the  SPEARS  CBT  T&E.  All  technical  issues  are  restated  here  for 
clarity. 
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Unless  otherwise  stated,  the  MOPs  were  collected  through  structured  protocols  conducted  in 
either  the  Aviation  Security  Laboratory  or  on-site  at  the  airport.  The  data  resulting  from  the 
technical  MOPs  include  the  answers  to  the  structured  interview  question  concerning  summary 
report  understandability,  annotated  results  from  the  review  of  pertinent  aspects  of  the  SPEARS 
device  against  HFE  checklists,  assessments  of  FTI  and  CTI  insertion  order  and  FTI  position 
assignment  within  bag  images,  image  control,  and  researcher  observations. 

MOP  data  for  each  technical  issue  were  assessed  against  its  related  criteria  to  ensure  compliance 
with  system  technical  requirements.  The  individual  SPEARS  technical  issues  checklists  will  be 
evaluated  to  ascertain  the  extent  that  these  criteria  have  been  met.  Responses  from  the  structured 
interviews  augmented  the  lED  test,  TIP  performance,  and  technical  issue  checklist  data. 


3.2.1  Technical  Issue  1 :  Image  Content. 


Do  images  represent  the  range  of  current  threats? 


Criterion  A.  At  least  300  different  CTIs  will  be  available  representing  threat  categories  of 
weapons,  incendiaries,  explosives,  and  other  dangerous/deadly  devices. 

Criterion  B.  At  least  300  different  FTIs  will  be  available  representing  threat  categories  of 
weapons,  incendiaries,  explosives,  and  other  dangerous/deadly  devices. 

Criterion  C.  At  least  two  different  aspect  angles  will  be  available  for  each  particular  threat 
object. 

Criterion  D.  The  degree  of  detection  difficulty  varies. 


3. 2. 1.1  Test  Strategy. 


An  inspection  of  the  threat  object  database  was  made  to  determine  whether  at  least  300  different 
CTIs  and  300  FTIs  are  available,  and  that  these  were  representative  of  the  threat  categories  of 
weapons,  incendiaries,  explosives,  and  other  dangerous/deadly  devices.  The  inspection  also 
determined  whether  at  least  two  different  aspect  angles  are  available  for  each  CTI  and  FTI  and 
that  the  degree  of  detection  difficulty  varied  for  these  different  items  in  CTI  displays  (  cf  the 
SPEARS  Image  Content  Checklist  in  Appendix  D). 


3.2.1 .2  MOPS  and  Data  Presentation. 
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MOP  9.  The  number  of  CTIs  available. 


The  Tn'P^  does  not  provide  CTIs. 

MOP  10.  The  threat  category  of  each  CTI  available. 

Not  demonstrated.  See  MOP  9. 

MOP  11.  The  number  of  FTIs  available. 

There  are  a  total  of  216  FTIs  available. 

MOP  12.  The  threat  category  of  each  FTI  available. 

Table  2  presents  the  number  of  FTIs  for  each  threat  category. 

TABLE  2  NUMBER  OF  FTIS  FOR  EACH  THREAT  CATEGORY 


FTI  Threat  Category 

#  of  FTIs 

Weapons 

96 

Explosives 

65 

Explosives  Components 

55 

MOP  13.  The  aspect  angles  available  for  each  CTI. 
Not  demonstrated.  See  MOP  9. 


MOP  14.  The  aspect  angles  available  for  each  FTI. 

At  least  two  different  aspect  angles  were  available  for  each  FTI  presentation. 


MOP  15.  Detection  difficulty  of  the  FTIs  varies  as  a  function  of  the  contents  of  the  actual 
bag  being  examined  and  will,  therefore,  not  be  assessed.  CTIs  were  assessed  by  ratings  provided 
by  screeners  or  subject-matter  experts. 
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Not  demonstrated.  See  MOP  9. 


3.2.1.3  Analysis  and  Discussion. 


This  issue  is  judged  not  met.  Since  the  Tn'P'^  could  not  display  CTIs,  all  MOPs  involving  CTIs 
could  not  be  evaluated.  The  system  did  present  FTIs  in  the  specified  categories  and  aspect 
angles;  however,  only  216  of  the  300  required  FTIs  were  available. 

3.2.2  Technical  Issue  2:  Customization. 


Can  performance  monitoring  be  tailored  to  individual  screener  needs? 

Criterion  A.  Images  can  be  selected  for  each  screener  as  a  function  of  threat  category,  difficulty, 
and  aspect  angle,  which  are  archived  along  with  the  associated  outcome  of  the  FTI  (hit,  miss, 
false  alarm,  correct  rejection). 

Criterion  R.  FTIs  and  CTIs  can  be  automatically  or  manually  selected  for  presentation  based  on 
such  factors  as  time  of  day,  checkpoint  activity,  or  screener  identity. 


3 .2.2.1  Test  Strategy. 


During  the  OT&E,  the  SPEARS  Customization  Checklist  (Appendix  E)  was  used  to  assess  the 
degree  to  which  TnT'’’’^  performance  monitoring  has  been  tailored  to  individual  screener  needs. 
The  checklist  items  covered  the  methods  for  selecting  and  archiving  of  CTIs  and  FTIs  using  both 
manual  and  automatic  processes. 


3. 2. 2.2  MOPS  and  Data  Presentation. 


MOP  16.  The  methods  of  selecting  CTIs  based  on  image  factors. 

Not  demonstrated.  See  MOP  9. 

MOP  17.  The  methods  of  archiving  results  for  CTIs  based  on  image  factors. 
Not  demonstrated.  See  MOP  9. 
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MOP  18.  The  methods  of  selecting  FTIs  based  on  image  factors. 


FTIs  cannot  be  customized  as  a  function  of  threat  category,  level  of  difficulty,  aspect  angle,  or  as 
a  combination  of  these  functions. 


MOP  19.  The  methods  of  archiving  results  for  FTIs  based  on  image  factors. 

FTIs  can  be  archived  with  the  associated  screener  response  outcome  (hit,  miss)  and  including  the 
date  and  time  of  day. 

MOP  20.  The  methods  of  selecting  CTIs  based  on  external  factors. 

Not  demonstrated.  See  MOP  9. 


MQP  2 1 .  The  methods  of  selecting  FTIs  based  on  external  factors. 

Although  FTIs  can  be  selected  automatically  based  on  time  of  day,  they  cannot  be  selected  based 
on  checkpoint  activity,  screener  identity,  or  as  a  combination  of  these  functions.  The  TnT^^ 
records  screener  FTI  detection  performance  and  maintains  this  information  in  performance 
reports.  Reports  are  date  and  time  stamped. 

3.2. 2. 3  Analysis  and  Discussion. 


This  issue  was  judged  not  met.  Again,  the  MOPs  related  to  CTIs  could  not  be  evaluated.  As  for 
FTIs,  the  system  did  not  permit  customization  of  the  image  selection  process  except  using  time 
of  day.  Customization  of  the  archiving  process  of  performance  reports  for  FTIs  was  limited  to 
external  factors  such  as  time  of  day. 


3-2.3  Technical  Issue  3:  Feedback. 


Is  feedback  provided? 


Criterion  A.  The  device  will  immediately  inform  the  screener  on  whether  each  threat  object  was 
correctly  identified. 
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3.2.3.1  Test  Strategy. 


Test  bags  not  containing  actual  threat  objects  were  passed  through  the  X-ray  scanning  device. 
The  SPEARS  device  was  used  to  insert  threat  object  images  onto  the  test  bag  images.  In 
addition,  test  bags  were  run  through  without  receiving  FTI  projections.  The  researcher  would 
correctly  or  incorrectly  press  the  TIP  key  when  an  image  is  presented  and  record  the  feedback 
provided.  For  the  purposes  of  this  test,  feedback  is  defined  as  any  combination  of  visual  or 
auditory  messages  that  is  intended  to  alert  the  X-ray  screener  and  explain  the  status  of  the  image 
scanning.  Additionally,  the  checklist  (Appendix  F)  was  used  to  determine  the  extent  to  which 
screener  feedback  is  provided. 


3. 2.3.2  MOPS  and  Data  Presentation. 


MOP  22.  The  feedback  provided  by  the  SPEARS  device  regarding  the  accuracy  of  threat 
object  disposition  as  determined  by  demonstration  and  inspection. 

The  TnT^^”  provides  feedback  to  the  screener  for  the  FTI  when  connected  to  the  Linescan  X-ray 
equipment.  Operator  feedback  is  displayed  in  a  consistent  position  and  also  has  a  consistent 
format  and  content.  Feedback  is  provided  for  correct  FTI  identification  and  for  threat  images, 
however,  the  feedback  is  slow  to  be  displayed.  Missed  threats  are  not  identified  and  the  FTI  is 
displayed  for  an  insufficient  time  before  it  is  removed. 

3. 2. 3. 3  Analysis  and  Discussion. 


This  issue  was  judged  not  met.  Although  feedback  is  provided  on  operator  correct  responses,  the 
system  is  slow  to  display  it.  More  importantly,  the  system  does  not  provide  feedback  on  missed 
threats,  which  is  critical  to  effective  training  of  screeners  to  detect  targets.  If  they  are  not  shown 
which  targets  were  missed,  they  can  never  achieve  the  additional  skill  level  required  to  detect 
more  difficult  threat  images.  The  lack  of  feedback  would  be  less  critical  if  screeners  were  at  a 
high  state  of  proficiency  prior  to  TIP  exposure. 

3.2.4  Technical  Issue  4:  Capability  Summaries. 


Are  performance  reports  prepared? 

Criterion  A.  The  device  automatically  generates  performance  reports  containing  descriptive 
statistics  and  readily  understandable  interpretations  to  summarize  records  of  threats  presented 
and  the  outcomes 
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Criterion  B.  Supervisors  can  be  alerted  when  screener  performance  falls  below  operational 
requirements  during  a  screening  session. 


3.2.4.1  Test  Strategy. 


The  performance  reports  were  automatically  tabulated  and  produced  by  the  SPEARS  device 
upon  prompting  by  the  researcher.  A  SPEARS  Capabilities  Summaries  Checklist  (Appendix  G) 
was  also  used  to  evaluate  the  content  and  presentation  methods  employed  in  the  performance 
summary  reports. 


3. 2.4.2  MOPS  and  Data  Presentation. 


MOP  23.  The  content  and  presentation  method  of  summary  reports  provided  by  the 
SPEARS  device  as  determined  by  demonstration  and  inspection. 

The  TnT™  reports  contain  screener  performance  scores  and  descriptive  statistics.  They  can  be 
presented  on  the  TnT'^“  monitor  and  printed.  The  Users  Log  Report  shows  a  date  and  time  stamp 
for  operator  log-on.  Some  operational  problems  were  noted,  however,  in  generating  reports  since 
some  of  them  take  up  to  30  minutes  to  generate  and  display  or  print  out.  During  this  time,  the 
computer  is  locked  to  the  operator,  who  cannot  perform  another  activities  with  the  system. 


MOP  24.  The  degree  to  which  reports  are  determined  to  be  understandable  through 
interviews  with  screeners  and  supervisors  and  assessment  using  HF  checklist  design  principles 
and  criteria. 

The  TnT™  manuals  provide  explanations  and  examples  of  the  various  reports  which  also  explain 
when  and  how  to  use  them.  The  headings  on  the  Users  Log  Report  are  easily  understandable 
except  for  the  third  column  which  is  labeled  only  with  a  “C”.  Labeling  the  column  as  “Class” 
would  be  more  intelligible. 


MOP  25.  The  performance  tracking  of  each  screener’s  missed  threat  totals  and  their 
transmission  to  supervisors  by  the  SPEARS  device  as  determined  by  description  and  inspection. 


Not  demonstrated.  See  paragraph  3.2.5  Interoperability 
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3.2.4.3  Analysis  and  Discussion. 


This  issue  was  judged  not  met.  The  system  did  not  automatically  warn  supervisor  of  inferior 
performance.  Although  some  reports  do  take  an  excessive  amount  of  time  to  generate,  few  other 
problems  were  noted  or  observed. 

3.2.5  Technical  Issue  5:  Interoperability. 


Can  the  TnT’’"'^  equipment  communicate  with  remote  computers? 

Criterion  A.  Additional  FTIs  and  CTIs  can  be  received  from  remote  sites. 
Criterion  B.  The  equipment  will  be  able  to  transmit  reports  to  remote  sites. 

3. 2. 5.1  Test  Strategy. 


The  Interoperability  Checklist  (Appendix  H)  was  used  to  evaluate  the  extent  to  which  the  TnT™ 
equipment  can  communicate  with  remote  computers.  A  computer  was  installed  at  a  remote 
location  and  connected  by  modem  with  the  device  at  the  test  site.  A  researcher  at  each  location 
attempted  to  download  new  images  to  the  SPEARS  devices.  Additionally,  researchers 
attempted  to  access  reports  from  the  remotely  situated  SPEARS  device. 


3. 2. 5. 2  MOPS  and  Data  Presentation. 


MOP  26.  The  method  of  remote  connection  to  the  SPEARS  device. 

The  TnT™  system  evaluated  was  not  capable  of  connecting  with  other  systems  in  a  network  and 
electronically  transmitting  data.  (See  also  MOP  25.)  The  same  was  true  for  the  system  used  at 
the  airport. 

MOP  27.  Demonstration  of  CTIs  downloaded  to  the  SPEARS  device. 

Not  demonstrated.  See  MOP  26 

MOP  28.  Demonstration  of  FTIs  downloaded  to  the  SPEARS  device. 
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Not  demonstrated.  See  MOP  26 


MOP  29.  Time  required  to  transfer  a  CTI. 

Not  demonstrated.  See  MOP  26 

MOP  30.  Time  required  to  transfer  an  FTI. 

Not  demonstrated.  See  MOP  26. 

MOP  31.  The  SPEARS  remote  performance  monitoring  capability. 
Not  demonstrated.  See  MOP  26 

3. 2. 5. 3  Analysis  and  Discussion. 


This  issue  was  judged  not  met  since  no  capability  was  demonstrated  for  remote  transmission. 


3.2.6  Technical  Is.sue  6:  Security. 


Is  access  restricted? 

Criterion  A.  Only  authorized  screeners  and  supervisors  can  access  certain  aspects  of  the  system. 


3.2.6. 1  Test  Strategy. 


The  security  issue  was  investigated  though  a  combination  of  a  structured  checklist  and  a 
demonstration  of  security  features.  The  security  checklist  is  based  on  the  Department  of  Defense 
Trusted  Computer  System  Evaluation  Criteria  (DOD-5200.28-STD)  and  is  contained  in 
Appendix  I. 
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3.2.6.2  MOPS  and  Data  Presentation. 


MOP  32.  Computer  Software  Security  Checklist  results. 

The  TuTtm  system  allows  only  authorized  operators  and  supervisors  to  access  certain  areas  of  the 
system.  Each  user  has  a  unique  identification  number  and  identifiable  password.  The  TnT™  does 
not  provide  sensitivity  labels  on  their  computer  files  because  the  system  limits  the  propagation  of 
sensitive  information  to  authorized  users  and  maintains  an  access  audit  trail.  Further,  the 
operating  system  is  password  protected,  which  eliminates  all  individuals  from  accessing  screener 
information  files  and  proprietary  software  files  assuming  the  maintenance  of  password 
confidentiality.  The  proprietary  software  files  are  encrypted  to  prevent  individuals  from  altering 
the  programs. 

3. 2.6. 3  Analysis  and  Discussion. 


This  issue  was  judged  Met.  The  system  is  capable  of  maintaining  files  with  restricted  access  to 
supervisors  and  requires  all  users  to  enter  an  appropriate  password. 

3.2.7  Technical  Issue  7:  Insertion. 


Can  FTI  images  be  automatically  and  unpredictably  (for  screeners)  inserted  on  the  checkpoint  X- 
ray  display?  Each  FTI  consists  of  only  a  single  threat  object. 

Criterion  A.  FTI  position  can  be  randomly  determined. 

Criterion  B.  FTI  orientation  within  each  bag  is  controllable. 


3.2.7. 1  Test  Strategy. 


Baggage  not  containing  threat  objects  were  passed  through  the  X-ray  device  and  the  SPEARS 
device  used  to  insert  FTIs  onto  the  display  of  the  bag  image.  For  the  bags  that  have  FTIs 
inserted,  the  position  of  each  threat  image  was  noted  on  a  3  by  3  cell  grid.  The  order  of 
placement  in  each  cell  across  FTIs  was  analyzed  for  randomness.  An  Image  Insertion  Checklist 
(Appendix  K)  was  used  to  evaluate  the  proper  insertion  of  images  into  the  checkpoint  X-ray 
display. 
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32.1.2  MOPS  and  Data  Presentation. 


MOP  33.  Recording  of  presentation  order  for  FTIs,  and  image  threat  placement  for  FTIs, 
using  a  baggage  content  locator  matrix,  which  will  be  used  to  determine  any  predictable 
presentation  patterns  and  FTI  orientation  controllability. 

The  Tn'P'^  was  able  to  generate  FTIs  (but  not  CTIs)  using  a  computer  generated  threat  object  and 
a  neutral  passenger  bag  passing  through  the  X-ray  system.  FTIs  were  generated  automatically, 
but  not  by  category.  They  had  no  apparent  predictability  of  presentation.  There  are  restrictions 
on  the  randomness  of  insertion  position  for  the  FTI  since  the  image  will  be  placed  only  where 
there  is  room  for  it  in  the  bag.  This  led  to  some  operational  problems  because  the  system 
occasionally  displayed  the  FTI  briefly  in  situations  where  the  bag  was  too  small  to  accommodate 
it.  The  FTI  was  then  dropped  from  the  bag  image  but  screeners  could  tell  that  the  system  had 
tried  to  insert  a  FTI  at  that  point.  However,  this  was  not  responded  to  nor  achieved. 


3.2.1 .3  Analysis  and  Discussion. 


This  issue  was  judged  Met  since  images  could  be  inserted  with  apparent  unpredictability  in  bag 
images  taking  into  account  obvious  physical  constraints  of  bag  and  FTI  size.  However,  the 
software  bug  which  permits  the  FTI  to  occasionally  be  displayed  while  trying  to  insert  it  into  a 
too  small  bag  needs  to  be  corrected. 

4^  CONCLUSIONS  AND  RF.COMMFNDATTONS 


4JL  SPEARS  TRATNTNO 


The  results  from  the  ANOVAs  and  post-hoc  analyses  indicated  that  the  TIP  training  produced  no 
significant  effect  on  training  group  performance  (  lED  Pj  P^,  d',  and  c)  on  either  of  the 
posttests.  Both  groups  showed  a  significant  rise  in  d’  over  the  course  of  the  three  tests  but  since 
the  control  group  received  no  TIP,  the  increase  must  be  ascribed  to  some  general  factor,  such  as 
LAX  raised  security  levels  around  this  time,  test  practice  or  increased  motivation,  rather  than 
from  the  training.  Post  hoc  analyses  performed  on  TIP  performance  in  the  training  group  showed 
a  one-time  jump  in  performance  between  weeks  2  and  3  of  the  training  and  then  a  stabilization  at 


32 


that  new  rate  for  the  rest  of  the  training.  The  results  demonstrate  that  the  training  as  provided  by 
the  system  had  no  discernible  effect  on  screener  detection  of  lEDs. 

4.2  SPEARS  VTGTT.ANCF,. 


A  one  time  jump  in  the  performance  of  the  training  group  was  observed  between  the  second  and 
third  weeks  of  FTl  exposure.  This  increase  was  sustained  but  not  increased  for  the  remainder  of 
the  treatments.  These  effects  are  consistent  with  an  increase  of  vigilance  caused  by  the  treatment 
effect,  however,  there  were  external  factors  which  provide  a  plausible  alternative  basis  for  the 
increase  in  hit  rate,  i.e.  the  increased  security  levels  during  that  period  of  time  at  LAX  due  to 
specific  threats.  In  addition,  both  the  training  and  the  control  groups  showed  increased  sensitivity 
on  the  first  and  second  posttests  compared  with  the  pretest.  An  account  based  on  FTI 
presentation  alone  cannot  explain  this  finding  since  only  the  training  group  received  the 
treatment. 


4.3  SPEARS  USABILITY  ASSESSMENT. 


Hardware,  software,  or  procedural  deficiencies  noted  are  reported  in  Appendix  B,  along  with  any 
HF  engineering  recommendations  for  rectification.  The  difficulty  in  linking  TIP  presentations  to 
particular  individuals  and  the  lack  of  timely  and  useful  performance  feedback  are  critical 
deficiencies  which  must  be  corrected.  Other  than  that,  the  TnT™  had  generally  acceptable 
usability  characteristics.  The  remaining  deficiencies  had  a  more  modest  impact  on  performance 
and  usability  of  the  system  including  no  Help  feature  and  poor  text  editing  and  menu  navigation 
attributes. 


4.4  TECHNICAL  TEST  AND  EV AT, HATTON 


The  TnT™  system  does  not  meet  the  majority  of  Technical  Criteria  established  for  it  in  the  TEP. 
Out  of  7  technical  issues,  the  TnT™  only  met  2  of  them:  Security  and  Image  Insertion.  The 
system  was  found  deficient  on  Image  Content,  Customization,  Feedback,  Capability  Summaries, 
and  Interoperability.  The  system  cannot  present  CTIs  and  had  only  216  of  the  300  required  FTIs 
were  available.  It  has  poor  feedback  properties,  no  capacity  for  interoperability  and  very  limited 
capacity  for  customization  of  FTI  selection  or  the  archiving  process. 
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OVERALL  CONrr.TTSTON 


In  its  present  configuration,  the  TnT'^*^  system  is  not  ready  for  operational  use  as  a  means  of 
providing  TIP  to  checkpoint  screeners.  Results  from  the  operational  test  at  LAX  and  from 
technical  testing  demonstrated  that  the  TnT™  system  did  not  meet  the  majority  of  approved 
critical  operational  and  technical  criteria  specified  in  the  Test  and  Evaluation  Plan.  The  group 
receiving  the  TIP  training  showed  no  differential  increase  in  screener  sensitivity  to  target 
detection  than  did  the  control  group  over  the  course  of  the  test  trials. 

These  results  indicate  that  it  is  not  useful  to  administer  TIP  under  conditions  of  limited 
accountability,  intermittent  exposure  to  the  treatment,  the  absence  of  demonstrated  mastery  of 
the  necessary  skills  to  detect  threats,  or  lack  of  knowledge  of  results.  The  inherent  benefits  of 
TIP  exposure  on  screener  performance  may  well  have  been  masked  by  the  difficulties  in 
providing  consistent  and  minimally  necessary  amounts  of  training  to  individuals  in  the  training 
group  as  well  as  the  other  test  constraints  and  system  limitations  described  in  the  report. 
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SIGNAL  DETECTION  THEORY  AND  APPLICATION 


SIGNAL  DETECTION  THEORY  rSDTI  PARADIGM. 

The  Improvised  Explosive  Device  Detection  System  operation  features  human  operators  engaged 
in  tasks  to  detect  an  environmental  event  or  signal.  SDT  is  a  mathematical  representation  of 
human  performance  in  deciding  whether  or  not  a  signal  is  present.  An  operational  example  of 
SDT  is  an  airport  security  guard  screening  passenger  bags  for  concealed  weapons  and 
Improvised  Explosive  Devices  (lEDs). 

There  are  two  response  categories  that  represent  a  screener’s  detection  performance:  Yes  (a 
Modular  Bomb  Set  [MBS]  signal  was  present)  or  No  (an  MBS  signal  was  not  present).  There  are 
also  two  signal  presentation  states  indicating  that  the  MBS  signal  was  present  (signal)  or  absent 
(noise).  A  combination  of  screener  responses  and  the  signal  state  produces  a  2  x  2  matrix  (figure 
^■1)>  generating  four  classes  of  operator  responses,  labeled  hits,  misses,  &lse  alarms,  and  correct 
rejections  (Wickens  1992).  Considering  the  Screener  Proficiency  Training  Evaluation  and 
Reporting  System  (SPEARS)  Test  and  Evaluation: 


State  of  MBS  Image 

MBS  Present  MBS  Not  Present 


Yes 


Screener 

Response 


No 


Hit 

False  Alarm 

Miss 

Correct 

Rejection 

FIGURE  A-1 .  2X2  MATRIX  OF  SCREENER  RESPONSES 
AND  STATE  OF  MBS  IMAGE 


a.  A  Hit  will  be  recorded  when  a  baggage  screener  correctly  detects  an  MBS  in  the 
scanned  baggage. 

b.  A  False  Alarm  will  be  recorded  when  a  baggage  screener  detects  an  MBS  in  the 
scanned  baggage  when  none  is  present. 
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As  indicated  by  Wickens  (1992),  the  SDT  paradigm  assumes  that  operators  perform  two  stages 
of  information  processing  in  all  detection  tasks:  (1)  sensory  evidence  is  aggregated  concerning 
the  presence  or  absence  of  the  signal,  and  (2)  a  decision  is  made  about  whether  this  evidence 
constitutes  a  signal.  According  to  SDT,  external  stimuli  generate  neural  activity  in  the  brain.  On 
the  average,  there  will  be  more  sensory  or  neural  evidence  in  the  brain  when  a  signal  is  present 
than  when  it  is  absent.  This  neiu^  evidence,  X,  referred  to  as  the  evidence  variable,  represents 
the  rate  of  firing  of  neurons  in  the  brain.  The  response  rate  for  detecting  X  increases  in 
magnitude  with  stimulus  (signal)  intensity.  Therefore,  if  there  is  enough  neural  activity,  X 
exceeds  a  critical  threshold,  X  ,  and  the  operator  decides  “yes.”  If  there  is  too  little,  the  operator 

C 

decides  “no.”  Because  the  amount  of  energy  in  the  signal  is  typically  low,  the  average  amount  of 
X  generated  by  signals  in  the  environment  is  not  much  greater  than  the  average  generated  when 
no  signals  are  present  (noise).  Furthermore,  the  quantity  of  X  varies  continuously,  even  in  the 
absence  of  a  signal,  because  of  random  variations  in  the  environment  and  the  operator’s  level  of 
neural  firing  (i.e.,  the  neural  “noise”  in  the  operator’s  sensory  channels  and  brain). 

The  relationship  between  the  presence  and  absence  of  a  signal  can  be  seen  in  the  hypothetical 
noise  and  signal  plus  noise  distributions  contained  in  figure  A-2.  The  intersection  of  the  two 
curves  represents  the  location  where  the  probability  of  a  signal  equals  the  probability  of  noise. 
The  criterion  value,  X  ,  chosen  by  the  operator,  is  shown  by  the  vertical  line.  All  X  values  to  the 

right  (X  >  X^  will  cause  the  operator  to  respond  “yes.”  All  X  values  to  the  left  generate  “no” 
responses. 


Title:  579D94-  1  .eps 

Creator:  CLARIS  EPSF  Export  Filter  V1  .0 
CreotionDote:  9/7/94  4:25:18  p.m. 


FIGURE  A-2.  HYPOTHETICAL  SDT  DISTRIBUTIONS  (Wickens  1992) 


The  different  shaded  areas  represent  the  occurrences  of  hits,  misses,  false  alarms,  and  correct 
rejections. 
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The  procedures  to  calculate  SDT  probabilities  are  as  follows: 

a.  In  SDT,  the  detection  values  are  expressed  as  probabilities. 

b.  The  probability  of  hit  (Ph),  miss  (PJ,  false  alarm  (P^^),  and  correct  rejection  (P^ 
are  determined  by  dividing  the  number  of  occurrences  in  a  cell  (figure  A-1)  by  the  total  number 
of  occurrences  in  a  column. 

c.  The  Pj^  (also  referred  as  the  probability  of  detection  [PJ)  will  be  calculated  by 
dividing  the  number  of  lEDs  detected  (number  of  hits)  by  the  total  number  of  hits  and  misses: 
P  =1-P,. 

m  a 

d.  The  P  will  be  determined  by  the  number  of  false  alarms  divided  by  the  total 

fa 

number  of  false  alarms  and  correct  rejections:  P^  =  1  -  P^,^. 

Operator  Response  Criterion. 

In  any  signal  detection  ta.<sk,  operator  decision  making  may  be  described  in  terms  of  an  operator 
response  criterion.  Operators  may  use  “risky”  response  strategies  by  responding  yes  more  often 
than  no.  A  risky  strategy  allows  operators  to  detect  most  of  the  signals  that  occur,  but  also 
produces  many  false  alarms.  Alternatively,  operators  may  use  “conservative”  strategies,  saying 
no  most  of  the  time,  making  few  false  alarms,  but  missing  many  of  the  signals. 

Different  circumstances  may  require  conservative  or  risky  strategies.  For  example,  an 
appropriate  TF.D  detection  strategy  requires  screeners  to  respond  “yes”  when  there  is  a  question 
regarding  baggage  contents.  This  response  may  produce  false  alarms  when  no  threatening 
objects  are  present. 

As  shown  in  figure  A-2,  risky  or  conservative  behavior  is  determined  by  the  location  of  the 
operator’s  response  criterion,  X^.  If  is  placed  to  the  right,  much  evidence  of  the  signal  is 

required  for  it  to  be  exceeded  and  most  responses  will  be  “no”  (conservative  responding).  If  it  is 
placed  to  the  left,  little  signal  evidence  is  required  and  most  responses  will  be  “yes,”  or  “risky.” 
A  variable  positively  correlated  with  X^  is  the  quantity  beta  ((3),  which  is  determined  as  follows: 


ORDh 

ORDfa 


(1) 


This  equation  is  the  ratio  of  the  ordinate  of  the  two  curves  of  figure  A-2  at  a  given  level  of  X^. 

The  higher  P  values  will  generate  fewer  yes  responses  and,  therefore,  fewer  hits.  Lower  P 
settings  will  generate  more  yes  responses,  more  hits,  and  more  false  alarms. 
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Table  A-1  provides  a  representative  table  of  Z  values  and  ordinate  values  of  the  probability 
distribution  related  to  hit  and  false  alarm  responses.  A  complete  table  of  standard  normal 
distribution  values  will  be  used  to  calculate  P  for  the  Test  and  Evaluation  Report  (TER). 


TABLE  A-1.  REPRESENTATIVE  Z-SCORES  AND  ORDINATE  VALUES  OF  THE 
NORMAL  CURVE  FOR  DIFFERENT  RESPONSE  PROBABILITIES 
TO  CALCULATE  p  AND  d' 


HIT/FA 

Z 

ORD 

HIT/FA 

Z 

ORD 

.01 

2.33 

0.03, 

.50 

0.00 

0.40 

.02 

2.05" 

0.05 

.55 

-0.12 

0.40 

.03 

1.88 

0.07 

.60 

-0.25 

0.39 

.04 

1.75 

0.09 

.65 

-0.38 

0.37 

.05 

1.64 

0.10 

.70 

-0.52 

0.35 

.08 

1.40 

0.15 

.75 

-0.67 

0.32 

.10 

1.28 

0.18 

.80 

-0.84 

0.28 

.13 

1.13 

0.21 

.82 

-0.92 

0.26 

.15 

1.04 

0.23 

.85 

-1.40 

0.23 

.18 

0.92 

0.26 

.88 

-1.18 

0.20 

.20 

0.84 

0.28 

.90 

-1.28 

0.18 

.25 

0.67 

0.32 

.92 

-1.40 

-0.15 

.30 

0.52 

0.35 

.95 

-1.64 

0.10 

.35 

0.38 

0.37 

.96 

-1.75 

0.09 

.40 

0.25 

0.39 

.97 

-1.88 

0.07 

.45 

0.12 

0.40 

.98 

-2.05 

0.05 

.50 

0.00 

0.40 

.99 

-2.33 

0.03 

The  procedures  required  to  calculate  P  are  listed  below  (Coren  and  Ward  1989): 

a.  Find  the  false  alarm  rate  from  the  outcome  matrix  in  the  HIT/FA  column  of  table 

A-1. 

b.  Read  across  the  table  to  the  ORD  column  (for  ordinate,  the  height  of  the  bell 

curve). 

c.  Determine  the  value  tabled  there  and  write  it  down. 

d.  Repeat  these  operations  for  the  hit  rate. 

e.  Calculate  p  using  the  following  equation:  P  =  ORDj^/ORD^^. 
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One  recent  parametric  measure  of  response  bias  is  £  (Ingham  1970;  Macmillan  &  Creelman 
1990;  and  Snodgrass  &  Corwin  1988).  The  chief  difference  between  the  measure  £  and  its 
parametric  alternative  P  lies  in  the  manner  in  which  they  locate  the  observer’s  criterion.  Whereas 
the  bias  index  P  locates  the  observer’s  criterion  by  the  ratio  of  the  ordinates  of  the  signal-plus  - 
noise  (SN)  and  noise  (N)  distributions,  £  locates  the  criterion  by  its  distance  from  the  intersection 
of  the  two  distributions  measured  in  z-score  units.  The  intersection  defines  the  point  where  bias 
is  neutral,  and  location  of  the  criterion  at  that  point  yields  a  £  value  of  0.  Conservative  criteria 
yield  positive  £  values,  and  liberal  criteria  produce  negative  £  values.  The  measme  £  is 
computed  as  follows: 


£  =  .5(Zfe  +  Zh) 


(2) 


Sensitivity  td'l. 

Sensitivity  refers  to  the  average  amount  of  operator  sensory  activity  generated  by  a  given  signal 
as  compared  with  the  average  amount  of  noise-generated  activity  (Coren  and  Ward  1989).  As 
explained  earlier,  baggage  screeners  may  fail  to  detect  (miss)  an  lED  signal  when  employing  a 
conservative  p.  Correspondingly,  the  signal  may  be  missed  because  the  resolution  of  the 
detection  process  is  low  in  discriminating  signals  from  noise,  even  if  P  is  neutral  or  risky. 

The  perceptual  analog  of  sensitivity,  d',  corresponds  to  the  separation  of  the  means  of  signal  and 
noise  distributions  (figure  A-2).  As  the  magnitude  of  the  signal  increases,  the  mean  of  the  signal 
distribution  moves  to  the  right.  The  proportion  of  signals  detected  (the  P^)  changes  as  the 

distance  between  the  signal  and  noise  distributions  varies.  According  to  Wickens  (1992),  if  the 
separation  between  the  distributions  is  great,  sensitivity  is  great,  an  operator  can  readily 
distinguish  a  signal  plus  noise  event  from  a  noise  only  event.  Similarly,  if  the  separation 
between  signal  and  noise  is  small,  d'  measures  will  be  low. 

Table  A-1  provides  a  representative  table  of  Z  values  and  ordinate  values  for  the  probability 
distribution  related  to  hit  and  false  alarm  responses.  A  complete  table  of  the  ordinate  values  for 
the  s^dard  normal  distribution  will  be  used  to  calculate  d'  for  the  TER. 

The  procedures  required  to  calculate  d'  are  listed  below  (McNicol  1972). 

a.  Separate  the  data  with  respect  to  distribution  (signal  or  noise),  and  total  the  data 
with  respect  to  response  (yes  or  no)  and  confidence  rating  (very  sure  or  not  very  sure). 

b.  Calculate?^  and  for  each  category. 

c.  Convert  the  values  of  P^  and  Pfg  to  Zj  and  z^  using  the  normal  curve  area. 

d.  Plot  Zj  against  z^  on  a  double-probability  plot  and  determine  the  path  of  the  ROC 
curve  based  upon  the  method  of  maximum  likelihood. 
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e.  Determine  the  ordinate  value,  for  either  axis,  at  which  the  ROC  curve  intersects 
with  the  negative  diagonal.  This  value  is  equal  to  /4d'.  Multiply  this  value  by  2  for  d  . 
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SPEARS  USABILITY  HFE  CHECKLIST 


Evaluation 

Human  Factors  Principle 

Acceptable 

Acceptable  with 
Deficiencies 

Unacceptable 

Comments 

1.0  Data  Entry 

1.  Users  need  enter  data  only  once. 

X 

2.  Display  feedback  for  all  user  actions 
during  data  entry;  display  keyed  entries 
stroke  by  stroke. 

X 

3.  Provide  fast  response  by  the  computer 
in  acknowledging  data  entries. 

X 

4.  Incorporate  a  consistent  method  for 
data  change. 

X 

5.  When  critical  data  are  to  be 
processed,  require  an  explicit  “Enter” 
action  to  initiate  the  processing. 

X 

6.  Provide  feedback  for  the  completion 
of  data  entry. 

X 

7.  For  repetitive  data  entry  transactions, 
feedback  should  be  the  generation  of  or 
moving  to  the  next  data  entry  field. 

NA 

8.  Choose  defining  abbreviations  or 
other  codes  to  shorten  data  entry  that  are 
distinctive  to  avoid  confusion  with  one 
another. 

NA 

9.  Allow  users  to  enter  each  character  of 
a  data  item  with  a  single  stroke  of  an 
appropriately  labeled  key. 

X 

10.  Treat  upper  and  lower  case  letters  as 
equivalent. 

X 
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Evaluation 

Human  Factors  Principle 

Acceptable 

Acceptable  with 
Deficiencies 

Unacceptable 

Comments 

11.  Treat  single  and  multiple  blank 
characters  as  equivalent  in  data  entry; 
do  not  require  users  to  count  spaces. 

X 

12.  For  position  designation  on  an 
electronic  display,  provide  a  movable 
cursor  with  distinctive  visual  features 
(shape,  blink,  etc.). 

X 

13.  Design  the  cursor  so  that  it  does  not 
obscure  any  other  character  displayed  in 
the  position  designated  by  the  cursor. 

X 

14.  Ensure  that  the  computer  will 
acknowledge  entry  of  a  designated 
position  quickly. 

X 

Trackball  needed 
double-click  to 
acknowledge  entry. 

15.  Ensure  that  the  displayed  cursor  will 
be  stable,  i.e.,  that  it  will  remain  where 
it  is  placed  until  moved  by  the  user  (or 
by  the  computer)  to  another  position. 

X 

16.  When  moving  a  cursor  from  one 
position  to  another,  design  the  cursor 
control  to  permit  both  fast  movement 
and  accurate  placement. 

X 

17.  Ensure  that  control  actions  for 
cursor  positioning  are  compatible  with 
movements  of  the  displayed  cursor,  in 
terms  of  control  function  and  labeling. 

X 

18.  If  a  cursor  must  be  positioned 
sequentially  in  redefined  areas,  such  as 
displayed  data  entry  fields,  ensure  that 
this  can  be  accomplished  by  simple  user 
action. 

X 

In  manager’s/add 
operation  sequence 
confusing  -  possible 
software  anomoly 
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Human  Factors  Principle 


Evaluation 


Acceptable  with 
Deficiencies 


19.  When  there  are  areas  of  a  display  in 

X 

which  data  entries  cannot  be  made 

(blank  spaces,  protected  field  labels, 

etc.),  make  those  areas  insensitive  to 

pointing  actions,  i.e.,  prevent  the  cursor 

from  entering  those  areas. 

Evaluation 

Human  Factors  Principle 

Acceptable 

Acceptable  with 
Deficiencies 

Unacceptable 

Comments 

20.  Ensure  that  an  ENTER  action  for 
multiple  data  items  results  in  entry  of  all 
items,  regardless  of  where  the  cursor  is 
placed  on  the  display. 

X 

21.  Ensure  that  display  capacity,  i.e., 
number  of  lines  and  line  length,  is 
adequate  to  support  eiSicient 
performance  of  tasks. 

X 

22.  When  text  has  been  specified  to 
become  the  subject  of  control  entries, 
highlight  that  segment  of  text  in  some 
way  to  indicate  its  boundaries. 

X 

Highlighting  used 
inconsistently 

23.  Unless  otherwise  specified  by  the 
user,  ensure  that  entered  text  is  left- 
justified,  leaving  right  margins  ragged  if 
that  is  the  result. 

X 

24.  Allow  users  to  select  and  move  text 
from  one  place  to  another. 

X 

No  cut/paste 
operation  available 

25.  Inform  users  concerning  the  status 
of  requests  for  printouts. 

NA 

No  printer  available 

26.  Design  text  editing  logic  so  that  any 
user  action  is  immediately  reversible. 

X 

27.  An  UNDO  action  should  be  able  to 
reverse  more  than  the  most  recent 
command. 

X 

No  undo  available  - 
Also  some  screen 
navigation  paths 
cannot  be  reversed. 

28.  Make  field  labels  consistent;  always 
employ  the  same  label  to  indicate  the 
same  kind  of  data. 

X 

29.  Protect  field  labels  from  keyed  entry. 

X 
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Evaluation 

Human  Factors  Principle 

Acceptable 

Acceptable  with 
Deficiencies 

Unacceptable 

Comments 

40.  When  a  measurement  unit  is 
consistently  associated  with  a  particular 
data  field,  include  that  unit  as  part  of  the 
field  label,  rather  than  requiring  a  user 
to  enter  it. 

X 

1 

41.  Employ  units  of  measurement  that 
are  familiar  to  the  user. 

X 

OK  for  US  users  - 
possible  problem  for 
foreign  users 

42.  Order  data  items  in  the  sequence  in 
which  a  user  will  think  of  them. 

X 

43.  The  order  of  data  items  should 
represent  a  logical  sequence  of  data 
entries. 

X 

44.  When  a  form  for  data  entry  is 
displayed,  the  computer  should  place  the 
cursor  automatically  at  the  beginning  of 
the  first  entry  field. 

X 

45.  When  sets  of  data  items  must  be 
entered  sequentially  in  a  repetitive 
series,  provide  a  tabular  display  format 
where  data  sets  can  be  keyed  row  by 
row. 

X 

X 

OK  on  some 
machines 

46.  Design  distinctive  formats  for 
column  headers  and  row  labels,  so  that 
users  can  distinguish  them  from  data 
entries. 

X 

47.  Ensure  that  column  headers  and  row 
labels  are  worded  informatively,  so  that 
they  will  help  guide  data  entry. 

X 

48.  During  tabular  data  entry,  allow 
users  to  tab  directly  from  one  data  field 
to  the  next,  so  that  the  cursor  can  move 
freely  back  and  forth. 

X 

ADD  OPERATORS 
-  some  fields  are 
omitted  from 
sequence 
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Evaluation 

Human  Factors  Principle 

Acceptable 

Acceptable  with 
Deficiencies 

Unacceptable 

Comments 

49.  When  a  user  must  enter  numeric 
values  that  will  later  be  displayed, 
maintain  all  significant  zeros;  zeros 
should  not  be  arbitrarily  removed  after  a 
decimal  point  if  they  affect  the  meaning 
of  the  number  in  terms  of  significant 
digits. 

NA 

50.  For  entry  of  tabular  data,  when 
entries  are  frequently  repeated,  provide 
users  with  some  easy  means  to  copy 
duplicated  data. 

X 

None  available 

51.  Provide  users  some  means  for 
designating  and  selecting  displayed 
graphic  elements  for  manipulation. 

When  a  user  has  changed  or  altered  a 
displayed  element,  provide  some 
indication  so  that  other  users  will  be 
aware  of  the  consequences  of  that  users 
actions. 

53.  Ensure  that  every  possible  correct 
data  entiy  will  be  accepted  and 
processed  properly  by  the  computer. 

X 

Not  all  features 
were  available 

54.  If  automatic  data  validation  detects  a 
probable  error,  display  an  error  message 
to  the  user.  If  data  validation  from  a 
secondary  source  detects  a  probable 
error,  allow  the  primary  user  access  to 
all  functions  required  to  check  data 
accuracy. 

! 

X 

55.  If  a  user  has  deferred  entry  of 
required  data,  but  then  requests 
processing  of  entries,  signal  that 
omission  to  the  user  and  allow 
immediate  entry  of  missing  items  or 
perhaps  further  deferral. 

X 

56.  If  a  set  of  default  values  have  been 
defined  for  a  data  entry  sequence,  allow 
a  user  to  accept  all  default  entries. 

NA 
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Evaluation 

Human  Factors  Principle 

Acceptable 

Acceptable  with 
Deficiencies 

Unacceptable 

Comments 

57.  When  interface  designers  cannot 
predict  what  default  values  will  be 
helpful,  permit  users  (or  a  system 
administrator)  to  define,  change,  or 
remove  default  values  for  any  data  entiy 
field. 

X 

Customize  TnT 
feature  was  disabled 
on  unit  evaluated 

58.  On  initiation  of  a  data  entry 
transaction,  display  currently  defined 
default  values  in  their  appropriate  data 
fields. 

X 

59.  Provide  users  with  some  simple 
means  to  confirm  acceptance  of  a 
displayed  default  value  for  entry. 

X 

60.  When  data  entries  made  in  one 
transaction  are  relevant  to  a  subsequent 
transaction,  program  the  computer  to 
retrieve  and  display  them  for  user 
review,  rather  than  requiring  re-entry  of 
those  data. 

X 

61.  Provide  automatic  cross-file 
updating  whenever  necessary,  so  that  a 
user  does  not  have  to  enter  the  same 
data  twice. 

X 

2.0  Data  Display 

1.  Ensure  that  whatever  data  a  user 
needs  for  any  transaction  will  be 
available  for  display. 

X 

2.  Tailor  displayed  data  to  user  needs, 
providing  only  necessary  and 
immediately  usable  data  for  any 
transaction;  do  not  overload  displays 
with  extraneous  data. 

X 

3.  Display  data  to  users  in  directly 
usable  form;  do  not  make  users  convert 
displayed  data. 

X 
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Evaluation 

Acceptable 

Acceptable  with 
Deficiencies 

Unacceptable 

Human  Factors  Principle 


Comments 


5.  For  any  particular  type  of  data 
display,  maintain  consistent  format  from 
one  display  to  another. 

X 

6.  Allow  users  to  change  displayed  data 
or  enter  new  data  when  that  is  required 
by  a  task. 

X 

7.  When  protection  of  displayed  data  is 
essential,  maintain  computer  control 
over  the  display  and  do  not  permit  a  user 
to  change  controlled  items. 

X 

8.  Ensure  that  each  data  display  will 
provide  needed  context,  recapitulating 
prior  data  as  necessary  so  that  a  user 
does  not  have  to  rely  on  memory  to 
interpret  new  data. 

X 

9.  The  wording  of  displayed  data  and 
labels  should  incorporate  familiar  terms 
and  the  task-oriented  jargon  of  the  users. 

X 

IMAGE 

CAPTURED  could 
be  confusing 

10.  Choose  words  carefully  and  then  use 
them  consistently. 

X 

11.  Ensure  that  wording  is  consistent 
from  one  display  to  another. 

X 

use  of  word  DONE 
in  Image  Library  is 
inconsistent 

12.  Use  consistent  grammatical  structure 
within  and  across  displays. 

X 

13.  When  abbreviations  are  used,  choose 
those  abbreviations  that  are  commonly 
recognized,  and  do  not  abbreviate  words 
that  produce  uncommon  or  ambiguous 
abbreviations. 

X 

‘c’  for  class  -  mm 
for  minutes 
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Evaluation 

Human  Factors  Principle 

Acceptable 

Acceptable  with 
Deficiencies 

Unacceptable 

Comments 

14.  Ensure  that  abbreviations  are 
distinctive,  so  that  abbreviations  for 
different  words  are  distinguishable. 

X 

15.  Minimize  punctuation  of 
abbreviations  and  acronyms. 

X 

16.  If  abbreviations  are  used,  provide  a 
dictionary/glossaiy  of  abbreviations 
available  for  on-line  user  reference. 

X 

No  online  glossary 

17.  When  a  critical  passage  merits 
emphasis  to  set  it  apart  from  other  text, 
highlight  that  passage  by  bolding, 
brightening,  color  coding,  or  some 
auxiliary  annotation. 

18.  Organize  data  in  some  recognizable 
order  to  facilitate  scanning  and 
assimilation. 

X 

19.  Display  continuous  test  in  wide 
columns,  containing  at  least  50 
characters  per  line. 

NA 

20.  Display  continuous  text 
conventionally  in  mixed  upper  and 
lower  case. 

X 

21.  Ensure  that  displayed  paragraphs  of 
text  are  separated  by  at  least  one  blank 
line. 

X 

22,  Maintain  consistent  spacing  between 
the  words  of  displayed  text,  with  left 
justification  of  lines  and  ragged  right 
margins  if  that  is  the  result. 

X 

23.  In  display  of  texlual  material,  keep 
words  intact,  with  minimal  breaking  by 
hyphenating  between  lines. 

X 
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Human  Factors  Principle 

Evaluation 

Acceptable 

Acceptable  with 
Deficiencies 

Unacceptable 

24.  Use  conventional  punctuation  in 
textual  display;  sentences  should  end 
with  a  period  or  other  special 
punctuation. 

X 

Comments 


24.  In  designing  text  displays,  especially 
text  composed  for  user  guidance,  strive 
for  simplicity  and  clarity  of  wording. 

X 

25.  Put  the  main  topic  of  each  sentence 
near  the  beginning  of  the  sentence. 

X 

26,  Use  short  simple  sentences. 

X 

27.  Use  affirmative  statements  rather 
than  negative  statements. 

X 

28.  Compose  sentences  in  the  active 
voice  rather  than  passive  voice. 

X 

29.  When  a  sentence  describes  a 
sequence  of  events,  phrase  it  with  a 
corresponding  word  order. 

X 

30.  Format  lists  so  that  each  item  starts 
on  a  new  line  (i.e.,  a  list  should  be 
displayed  as  a  single  column). 

X 

31.  When  a  single  item  in  a  list 
continues  for  more  than  one  line,  mark 
items  in  some  way  so  that  the 
continuation  of  an  item  is  obvious  (i.e., 
so  that  a  continued  portion  does  not 
appear  to  be  a  separate  item). 

X 

32.  When  listed  items  will  be  numbered, 
use  Arabic  rather  than  Roman  numerals. 

X 
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Evaluation 

Human  Factors  Principle 

Acceptable 

Acceptable  with 
Deficiencies 

Unacceptable 

Comments 

33.  Adopt  some  logical  principle  by 
which  to  order  lists.  When  no  other 
principle  applies,  order  lists 
alphabetically. 

X 

34.  If  a  list  is  displayed  in  multiple 
columns,  order  the  items  vertically 
within  each  cell. 

X 

35.  For  a  long  list,  extending  more  than 
one  displayed  page,  consider  adopting  a 
hierarchic  structure  to  permit  its  logical 
partitioning  into  related  shorter  lists. 

NA 

36.  When  words  in  text  displays  are 
abbreviated,  define  each  abbreviation  in 
parentheses  following  its  first 
appearance. 

NA 

None 

37.  When  text  is  combined  with 
graphics  or  other  data  in  a  single 
display,  thus  limiting  the  space  available 
for  text,  format  the  text  in  a  few  wide 
lines  rather  than  in  narrow  columns  of 
many  short  lines. 

X 

38.  Separate  the  data  in  a  table  by  some 
distinctive  feature,  to  ensure  separations 
of  entries  within  a  row. 

X 

39.  In  dense  tables  with  many  rows, 
insert  a  distinctive  feature  to  aid 
horizontal  scanning  at  regular  intervals. 

X 

40.  Ensure  that  row  and  column  labels 
are  distinguishable  from  the  data 
displayed  within  tables  and  from  the 
labels  of  displayed  lists,  such  as  menu 
options  or  instructions  to  users. 

X 

41.  When  rows  or  columns  are  labeled 
by  number,  start  the  numbering  with 
‘‘1,”  rather  than  “0.” 

X 
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Evaluation 

Human  Factors  Principle 

Acceptable 

Acceptable  with 
Deficiencies 

Unacceptable 

Comments 

42.  Show  columns  of  alphabetic  data 
with  left  justification  to  permit  rapid 
scanning. 

X 

43.  Justify  columns  of  numeric  data  with 
respect  to  a  fixed  decimal  point;  if  there 
is  no  decimal  point,  then  numbers 
should  be  right  justified. 

X 

44,  Use  consistent  logic  in  the  design  of 
graphic  displays,  and  maintain  standard 
format,  labeling,  etc. 

X 

45.  Tailor  graphic  displays  to  user  needs 
and  provide  only  those  data  necessary 
for  user  tasks. 

X 

46.  When  graphics  contain  outstanding 
or  discrepant  features  that  merit 
attention  by  a  user,  consider  displaying 
supplementary  text  to  emphasize  that 
feature. 

X 

47.  When  a  user’s  attention  must  be 
directed  to  a  portion  of  a  display 
showing  critical  or  abnormal  data, 
highlight  that  feature  with  some 
distinctive  means  of  coding. 

X 

No  use  of  effective 
highlight 

48.  Adopt  a  consistent  organization  for 
the  location  of  various  display  features 
from  one  display  to  another. 

X 

49,  Make  the  different  elements  of  a 
display  format  distinctive  from  one 
another. 

X 

50,  Use  blank  space  to  structure  a 
display. 

X 
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Evaluation 

Human  Factors  Principle 

Acceptable 

Acceptable  with 
Deficiencies 

Unacceptable 

Comments 

51.  Begin  eveiy  display  at  the  top  with  a 
title  or  header,  describing  briefly  the 
contents  or  purpose  of  the  display. 

X 

52.  Reserve  the  last  several  lines  at  the 
bottom  of  every  display  for  status  and 
error  messages,  prompts,  and  cottunand 
entry. 

X 

Not  done 

53,  Ensure  that  displays  are  formatted  to 
group  data  items  on  the  basis  of  some 
logical  principle. 

X 

54.  Provide  distinctive  coding  to 
highlight  important  display  items 
requiring  user  attention.  Highlighting  is 
most  effective  when  used  sparingly. 

X 

No  effective  use  of 
highlighting 

55.  If  highlighting  is  used  to  emphasize 
important  display  items,  remove  such 
highlighting  when  it  no  longer  has 
meaning. 

X 

56.  Adopt  meaningful  or  familiar  codes, 
rather  than  arbitrary  codes. 

X 

57.  Adopt  codes  for  display  (and  entry) 
that  conform  with  accepted 
abbreviations  and  general  user 
expectations. 

X 

58.  Assign  consistent  meanings  to 
symbols  and  other  codes,  from  one 
display  to  another. 

X 

59.  Treat  brightness  as  a  two-valued 
code,  bright  and  dim,  i.e„  consider 
coding  by  difference  in  brightness  for 
applications  that  only  require 
discriminating  between  two  categories 
of  displayed  items. 

X 

Not  effectively  used 
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Evaluation 
Acceptable  with 
Deficiencies 


Comments 


Human  Factors  Principle 


Acceptable 


Unacceptable 


60.  Color  code  for  relative  values. 


Colors  are 
meaningless  on 
graphs 


61.  It  will  prove  safer  to  use  no  more 
than  five  different  colors  for  coding. 


62.  When  selecting  colors  for  coding 
discrete  categories  of  data,  ensure  that 
those  colors  are  easily  discriminable. 


63.  Employ  color  coding  conservatively, 
using  relatively  few  colors  and  only  to 
designated  critical  categories  of 
displayed  data. _ 


X 


X 


X 


64.  Make  color  coding  redundant  with 
some  other  display  feature,  such  as 
symbology;  do  not  code  only  by  color. 

X 

65.  When  color  coding  is  used,  ensure 
that  each  color  represents  only  one 
category  of  displayed  data. 

X 

66.  Choose  colors  for  coding  based  on 
conventional  associations  with 
particular  colors. 

X 

67.  Use  brighter  and/or  more  saturated 
colors  when  it  is  necessary  to  draw  a 
user’s  attention  to  critical  data. 

X 

Not  used 

68.  Use  saturated  blue  only  for 
background  features  in  a  display,  and 
not  for  critical  coding. 

X 

69.  When  blink  coding  is  used,  select  a 
blink  rate  in  the  rage  from  2  to  5  Hz, 
with  a  minimum  duty  cycle  (ON 
interval)  of  50  percent. 

NA 
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2.  Employ  similar  means  to  accomplish 
ends  that  are  similar,  from  one 
transaction  to  the  next,  from  one  task  to 
another,  throughout  the  user  interface. 

X 

3.  Display  some  continuous  indication  of 
current  context  for  reference  by  the  user. 

X 

4.  Adopt  consistent  terminology  for  on¬ 
line  guidance  and  other  messages  to 
users. 

X 

5.  Choose  names  that  are  semantically 
congruent  with  natural  usage,  especially 
for  paired  opposites  (e.g.,  UP  /  DOWN). 

X 

6.  Ensure  that  the  computer 
acknowledges  every  entiy  immediately; 
for  every  action  by  the  user  there  should 
be  some  apparent  reaction  from  the 
computer. 

X 

Feedback  on 
operators  first 
screen  is  missing 

7.  Each  menu  display  should  permit 
only  one  selection  by  the  user. 

X 
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12.  Design  a  menu  to  display  only  those  X 
options  that  are  actually  available  in  the 
current  context  for  a  particular  user. 

13.  When  menus  are  provided  in  X 

different  displays,  design  them  so  that 
option  lists  are  consistent  in  wording 

and  ordering. 

14.  If  menu  options  are  included  in  a  X 
display  that  is  intended  for  data  review 
and/or  data  entry,  ensure  that  they  are 
distinct  from  other  displayed 
information;  incorporate  some 

consistent  distinguishing  feature  to 
indicate  their  special  function. 


15.  List  displayed  menu  options  in  a  X 
logical  order;  if  no  logical  structure  is 
apparent,  then  display  the  options  in  the 
order  of  their  expected  frequency  of  use, 
with  the  most  frequent  listed  first. 
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Human  Factors  Principle 


Evaluation 


Acceptable  with 
Deficiencies 


Comments 


16.  Format  a  menu  to  indicate  logically 
related  groups  of  options,  rather  than  as 
an  undifferentiated  string  of 
alternatives. 

NA 

17.  When  the  menu  selection  must  be 
made  from  a  long  list,  and  not  all 
options  can  be  displayed  at  once, 
provide  a  hierarchic  sequence  of  menu 
selections,  rather  than  one  long 
multipage  menu. 

NA 

18.  When  hierarchic  menus  are  used, 
design  their  structure  to  permit 
immediate  user  access  to  critical  or 
frequently  selected  options. 

NA 

19.  When  hierarchic  menus  are  used, 
display  some  indication  of  current 
position  in  the  menu  structure. 

NA 

20.  When  hierarchic  menus  are  used,  NA 
ensure  that  display  format  and  selection 

logic  are  consistent  at  every  level. 

21.  When  hierarchic  menus  are  used,  NA 
require  users  to  take  only  one  simple  key 
action  to  return  to  the  next  higher  level. 

22.  When  hierarchic  menus  are  used,  NA 
require  users  to  take  only  one  simple  key 
action  to  return  to  the  general  menu  at 

the  top  level. 

23.  Design  the  general  options  list  to  NA 
show  control  entry  options  grouped, 
labeled,  and  ordered  in  terms  of  their 

logical  function,  and  frequency  and 
criticality  of  use. 

24.  Make  available  to  users  a  list  of  the  NA 
control  options  that  are  specifically 
appropriate  for  any  transaction. 
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Evaluation 


Human  Factors  Principle 

Acceptable 

Acceptable  with 
Deficiencies 

Unacceptable 

Comments 

25.  Offer  users  only  control  options  that 
are  actually  available  for  the  current 
transactions. 

X 

26.  When  a  user  is  performing  an 
operation  on  some  selected  display  item, 
highlight  that  item. 

X 

Highlighting  rarely 
used 

27.  Design  the  interface  software  to  deal 
appropriately  with  all  possible  control 
entries,  correct  and  incorrect. 

X 

28.  When  a  user  completes  correction  of 
an  error,  require  the  user  to  take  an 
explicit  action  to  reenter  the  corrected 
material;  use  the  same  action  for  reentry 
that  was  used  for  the  original  entry. 

X 

29.  When  a  control  entry  will  cause  any 
extensive  change  in  stored  data, 
procedures,  and/or  system  operation, 
and  particularly  if  that  change  cannot  be 
easily  reversed,  notify  the  user  and 
require  confirmation  of  the  action  before 
implementing  it.  Provide  a  prompt  to 
confirm  actions  that  will  lead  to  possible 
data  loss. 

X 

All  features  not 
available 

30.  Word  the  prompt  for  a  CONFIRM 
action  to  warn  users  explicitly  of  any 
possible  data  loss. 

31.  Ensure  that  any  user  action  can  be 
immediately  reversed  by  an  UNDO 
command. 

32.  UNDO  itself  should  be  reversible,  so 
that  a  second  UNDO  action  will  do 
again  whatever  was  just  undone. 
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Human  Factors  Principle 

Evaluation 

Comments 

Acceptable 

Acceptable  with 
Deficiencies 

Unacceptable 

33.  When  a  user  requests  LOG-OFF, 
check  pending  transactions  and,  if  any 
pending  transaction  will  not  be 
completed  or  if  data  will  be  lost,  display 
an  advisory  message  requesting  user 
confirmation.  Ensure  that  the  user  is 
aware  of  the  status  of  the  system  or  any 
of  its  components  after  LOG-OFF  is 
complete. 

4.0  User  Guidance 

1.  Design  standard  procedures  for 
accomplishing  similar,  logically  related 
transactions. 

2.  Require  users  to  take  explicit  actions 
to  specily  computer  processing;  the 
computer  should  not  take  extra  actions 
beyond  those  specified  by  a  user. 

i 

3.  In  applications  where  users  must  log 
on  to  the  system,  design  LOG-ON  as  a 
separate  procedure  that  is  completed 
before  a  user  is  required  to  select  among 
any  operational  options. 

4.  Ensure  that  only  relevant  data  are 
displayed  by  tailoring  the  display  for  any 
transaction  to  the  current  information 
requirements  of  the  user. 

5.  Create  display  formats  with  a 
consistent  structure  evident  to  the  user, 
so  that  a  particular  type  of  data  is  always 
presented  in  the  same  place  and  in  the 
same  way. 

6.  Format  each  different  type  of  user 
guidance  consistently  across  displays. 
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Evaluation 

Acceptable 

Acceptable  with 
Deficiencies 

Unacceptable 

Comments 

7.  Design  display  formats  so  that  user 
guidance  material  is  readily 
distinguishable  from  displayed  data. 

8.  Design  cursors  so  that  they  are  readily 
distinguished  from  other  displayed 
items. 

9.  Label  all  displayed  data  clearly. 

10.  Ensure  that  symbols  and  other  codes 
have  consistent  meanings  from  one 
display  to  another. 

1 1.  Ensure  that  the  names  for  functions 
and  data  are  consistent  for  similar  or 
identical  functions. 

12.  Adopt  wording  terminology  familiar 
to  users. 

13.  Adopt  task-oriented  wording. 

14.  Adopt  affirmative  rather  than 
negative  wording  for  user  guidance 
messages. 

15.  Adopt  active  rather  than  passive 
voice  in  user  guidance  messages. 

16.  When  user  guidance  describes  a 
sequence  of  steps,  follow  that  same 
sequence  in  the  wording  of  user 
guidance. 

17.  Be  consistent  in  grammatical 
construction  when  wording  user 
guidance. 

18.  Allow  users  to  switch  easily  between 
information  handling  and  associated 
guidance  material. 
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Evaluation 

Comments 

Acceptable 

Acceptable  with 
Deficiencies 

Unacceptable 

19.  Provide  some  indication  of  system 
status  to  users  at  all  times. 

20.  When  users  must  log  on  to  a  system, 
display  appropriate  prompts  for  LOG¬ 
ON  procedures  automatically  at  a  user’s 
terminal. 

21.  If  a  user  tries  to  log  onto  a  system 
and  LOG-ON  is  denied  because  of 
system  unavailability,  display  an 
advisory  message  telling  the  user  what 
the  system  status  is  and  when  the  system 
will  become  available. 

22.  When  task  performance  requires 
data  exchange  and/or  interaction  with 
other  users,  allow  a  user  to  obtain  status 
information  concerning  other  people 
currently  using  the  system. 

23.  Ensure  that  every  input  by  a  user 
will  consistently  produce  some 
perceptible  response  output  from  the 
computer. 

24.  Ensure  that  computer  response  to 
user  entries  will  be  rapid,  with 
consistent  timing  as  appropriate  for 
different  types  of  transactions. 

25.  Provide  some  indication  of 
transaction  status  whenever  the 
complete  response  to  a  user  entry  will  be 
delayed. 

26.  Provide  a  unique  identification  for 
each  display  in  a  consistent  location  at 
the  top  of  the  display  frame. 
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Evaluation 

Acceptable 

Acceptable  with 
Deficiencies 

Unacceptable 

Comments 

27.  When  a  user  (or  computer)  action 
establishes  a  change  in  operational 
mode  that  will  affect  subsequent  user 
actions,  display  some  continuing 
indication  of  current  mode. 

28.  When  the  computer  detects  an  entry 
error,  display  an  error  message  to  the 
user  stating  what  is  wrong  and  what  can 
be  done  about  it. 

29.  Make  the  wording  of  error  messages 
as  specific  as  possible. 

30.  Make  error  messages  brief  but 
informative. 

31.  Adopt  neutral  wording  for  error 
messages;  do  not  imply  blame  to  the 
user,  or  personalize  the  computer,  or 
attempt  to  make  a  message  humorous. 

32.  The  computer  should  display  an 
error  message  only  after  a  user  has 
completed  an  entry. 

33.  Display  an  error  message  a 
minimum  of  2  to  4  seconds  after  the 
user  entry  in  which  the  error  is  detected. 


34.  As  a  supplement  on-line  guidance, 
include  in  the  system  documentation 
listing  and  explaining  all  error 
messages. 


35.  In  addition  to  providing  an  error 
message,  mark  the  location  of  a  detected 
error  by  positioning  the  cursor  at  that 
point  on  the  display,  i.e.,  at  that  data 
field  or  command  word. 
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36.  When  an  entry  error  has  been 
detected,  continue  to  display  the 
erroneous  entiy,  as  well  as  an  error 
message,  until  corrections  are  made. 

37.  Following  error  detection,  require 
the  user  to  reenter  only  that  portion  of  a 
data/command  entry  which  is  not 
correct. 


38.  Ensure  that  a  displayed  error 
message  is  removed  after  the  error  has 
been  corrected;  do  not  continue  to 
display  a  message  that  is  no  longer 
applicable. 

39.  Ensure  that  specific  user  guidance 
information  is  available  for  display  at 
any  point  in  a  transaction  sequence. 

40.  Display  menu  options  in  logical 
groups. 


41.  Provide  reference  material 
describing  system  capabilities  and 
procedures  available  to  users  for  on-line 
display. 

42.  In  addition  to  explicit  and  implicit 
aids,  permit  users  to  obtain  further  on- 
line  guidance  by  requesting  HELP. 

43.  Provide  a  simple,  standard  action 
that  is  always  available  to  request 
HELP. 


44.  When  an  initial  HELP  display 
provides  only  summary  information, 
provide  more  detailed  explanations  in 
response  to  repeated  user  requests  for 
HELP. 


Human  Factors  Principle 

Evaluation 

Acceptable 

Acceptable  with 
Deficiencies 

Unacceptable 

Comments 

45.  Permit  users  to  browse  through  on¬ 
line  HELP  displays,  just  as  they  would 
through  a  printed  manual,  to  gain 
familiarity  with  system  functions  and 
operating  procedures. 

5.0  Data  Transmission 

1.  Choose  functional  wording  for  terms 
used  in  data  transmission,  including 
messages,  for  initiating  and  controlling 
message  transmission  and  other  forms  of 
data  transfer,  and  for  receiving 
messages. 

2.  Design  the  data  transmission 
procedures  to  minimize  memory  load  on 
the  user. 

3.  Design  the  data  transmission 
procedures  to  minimize  required  user 
actions. 

4.  Design  the  data  transmission 
procedures  so  that  both  sending  and 
receiving  messages  are  accomplished  by 
explicit  user  action. 

5.  Allow  users  to  interrupt  message 
preparation,  review,  or  disposition,  and 
then  resume  any  of  those  tasks  from  the 
point  of  interruption. 

6.  Provide  software  capabilities  to 
annotate  transmitted  data  with 
appropriate  highlighting  to  emphasize 
alarm/alert  conditions,  priority 
indicators,  or  other  significant  second- 
order  information  that  could  affect 
message  handling. 
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Evaluation 

Comments 

Acceptable 

Acceptable  with 
Deficiencies 

Unacceptable 

7.  For  addressing  and  identifying 
messages,  provide  a  basic  set  of  header 
fields  that  can  be  interpreted  by  all 
systems  to  which  users  will  send 
messages. 

8.  Allow  users  to  print  copies  of 
transmitted  messages  in  order  to  make 
hard-copy  records. 

9.  Ensure  that  transmitted  data  are 
protected  automatically  with  parity 
checks  to  detect  and  correct  any  errors 
that  may  occur,  and  buffering  until 
acknowledgment  of  receipt. 

10.  Ensure  that  only  one  copy  of  a 
message  will  be  transmitted  to  any 
individual  addressee. 

11.  For  receiving  messages,  allow  users 
to  choose  the  method  of  receipt  (i.e., 
files,  display,  printer)  that  will  be  the 
local  destination. 

12.  Ensure  that  computer  aids  and 
procedures  for  reviewing  messages  are 
consistent  with  other  system  capabilities 
for  general  data  display. 

6.0  Data  Protection 

1.  Provide  automatic  measures  to 
minimize  data  loss  from  computer 
failure. 

2.  Protect  data  from  inadvertent  loss 
caused  by  the  actions  of  other  users. 
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3,  Provide  clear  and  consistent 
procedures  for  different  types  of 
transactions,  particularly  those 
involving  data  entry,  change  and 
deletion,  and  error  correction. 


4.  Ensure  that  the  ease  of  user  actions 
will  match  desired  ends;  make  frequent 
or  urgent  actions  easy  to  take,  but  make 
potentially  destructive  actions 
sufficiently  difficult  that  they  will 
require  extra  user  attention. 


5.  Ensure  that  the  computer  changes 
data  only  as  a  result  of  explicit  actions 
by  a  user,  and  does  not  initiate  changes 
automatically. 

6.  For  conditions  which  may  require 
special  user  attention  to  protect  against 
data  loss,  provide  an  explicit  alarm 
and/or  warning  message  to  prompt 
appropriate  user  action. 


7.  Allow  users  to  UNDO  an  immediately 
preceding  control  action  that  may  have 
caused  an  unintended  data  loss. 


8.  Design  the  LOG-ON  process  and 
procedures  for  user  identification  to  be 
as  simple  as  possible  and  consistent  with 
protecting  data  from  unauthorized  use. 


9.  Design  the  LOG-ON  process  to 
provide  prompts  for  all  user  entries, 
including  passwords  and/or  whatever 
other  data  are  required  to  confirm  user 
identity  and  to  authorize  appropriate 
data  access/change  privileges. _ 


10.  When  passwords  are  required,  allow 
users  to  choose  their  own  passwords. 


_ Human  Factors  Principle _ 

11.  Allow  users  to  change  their 
passwords  whenever  they  choose. 

12.  When  a  password  must  be  entered  by 
a  user,  ensure  that  password  entry  can 
be  private;  password  entries  should  not 
be  displayed. 

13.  Impose  a  maximum  limit  on  the 
number  and  rate  of  unsuccessful  LOG¬ 
ON  attempts  that  will  provide  a  margin 
for  user  error  while  protecting  the 
system  from  persistent  attempts  at 
illegitimate  access. 

14.  Once  a  user’s  identity  has  been 
authenticated,  ensure  that  whatever  data 
access/change  privileges  are  authorized 
for  that  user  will  continue  throughout  a 
work  session. 


15.  Establish  user  authorization  for  data 
access  at  initial  LOG-ON;  do  not  require 
further  authentication  when  a  user 
requests  display  of  particular  data. 

16.  When  displayed  data  are  classified 
for  security  purposes,  include  a 
prominent  indication  of  security 
classification  in  each  display. 

17.  When  confidential  information  is 
displayed  at  a  work  station  that  might  be 
viewed  by  casual  onlookers,  provide  the 
user  with  some  rapid  means  of 
temporarily  suppressing  a  current 
display,  and  then  resuming  work  later. 

18.  When  data  must  not  be  changed, 
maintain  computer  control  over  the  data, 
and  do  not  permit  users  to  change 
controlled  items. 


Human  Factors  Principle 

Evaluation 

Comments 

Acceptable 

Acceptable  with 
Deficiencies 

Unacceptable 

19.  Make  procedures  for  data 
entry/change  as  simple  as  possible. 

20.  Allow  users  to  enter  logically  related 
items  with  a  single,  explicit  action  at  the 
end  of  the  sequence,  rather  than 
entering  each  item  separately. 

21.  When  a  data  entry  error  is  detected 
by  the  computer,  allow  the  user  to  make  i 
an  immediate  correction. 

22.  Following  error  detection,  allow 
users  to  edit  entries  so  that  they  must 
rekey  only  those  portions  that  were  in 
error. 

23.  Require  users  to  take  explicit  action 
to  confirm  doubtful  and/or  potentially 
destructive  data  change  actions  before 
they  are  accepted  by  the  computer  for 
execution. 

24.  When  a  user  requests  LOG-OFF, 
check  pending  transactions  involving 
data  entry/change  and,  if  data  loss  seems 
probable,  display  an  appropriate 
advisory  message  to  the  user. 
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7.0  Hardware  (MIL-STD  1472D) 

Visual  Displays 

1.0  Contrast.  Sufficient  contrast  shall 
be  provided  between  displayed 
information  and  the  display  background 
to  ensure  that  the  required  information 
c^n  be  perceived  by  the  operator  under 
all  expected  lighting  conditions. 

2.0  Location.  Displays  shall  be  located 
and  designed  so  that  they  may  be  read  to 
the  degree  of  accuracy  required  by 
j)ersonnel  in  normal  operating  or 
servicing  positions  without  requiring  the 
operator  to  assume  an  uncomfortable, 
awkward,  or  unsafe  position. 

3.0  Orientation.  Display  faces  shall  be 
perpendicular  to  the  operator’s  normal 
line  of  sight  whenever  feasible  and  shall 
not  be  less  than  3.14/4  rad  (45°)  from 
the  normal  line  of  sight. 

4.0  Reflection.  Displays  shall  be 
constructed,  arranged,  and  mounted  to 
prevent  reduction  of  information 
transfer  due  to  the  reflection  of  the 
ambient  illumination  from  the  display 
cover.  Reflection  of  instruments  and 
consoles  and  other  enclosures  shall  be 
avoided. 


5.0  Frequency  of  use.  Displays  used 
most  frequently  should  be  grouped 
together  and  placed  in  the  optimum 
visual  zone. 


CRTs 


Human  Factors  Principle 


Acceptable 


Evaluation 
Acceptable  with 
Deficiencies 


Unacceptable 


Comments 


1.0  Viewing  distance.  A  400  mm  (16- 
inch)  viewing  distance  shall  be  provided 
whenever  practicable.  When  periods  of 
observation  will  be  short,  or  when  dim 
signals  must  be  detected,  the  viewing 
distance  may  be  reduced  to  250  mm  (10 
inches).  Design  should  permit  the 
observer  to  view  the  CRT  as  close  as 
desired. 


2.0  Reflected  glare.  Reflected  glare 
shall  be  minimized  by  proper  placement 
of  the  CRT  relative  to  the  light  source 
and  the  use  of  a  hood,  shield,  or  optical 
coating  on  the  CRT. 


3.0  Ambient  surfaces.  Surfaces 
adjacent  to  the  CRT  shall  have  a  dull 
matte  finish. 


4.0  Font  legibility.  Where 
alphanumeric  characters  appear  on 
CRT-like  displays,  the  font  style  shall 
allow  discrimination  of  similar 
characters,  such  as  the  letter  1  and  the 
number  1  and  the  letter  z  and  the 
number  2. 


Audio  Displays 


1,0  Type  of  voice.  The  voice  used  in 
recording  verbal  signals  shall  be 
distinctive  and  mature. 


2,0  Delivery  style.  Verbal  signal  shall 
be  presented  in  a  formal,  impersonal 
manner. 


B-32 


Human  Factors  Principle 

Evaluation 

Comments 

Acceptable 

Acceptable  with 
Deficiencies 

Unacceptable 

3.0  Message  content.  In  selecting 
words  to  be  used  in  audio  warning 
signals,  priority  shall  be  given  to 
intelligibility,  aptness,  and  conciseness 
in  that  order. 

4.0  Volume  control  (automatic  or 
manual).  The  volume  (loudness)  of  an 
audio  warning  signal  shall  be  designated 
to  be  controlled  by  the  operator,  the 
sensing  mechanism,  or  both. 

Keyboards 

1.0  Numeric  keyboard.  The 
configuration  of  a  keyboard  used  to 
enter  solely  numeric  information  should 
be  a  3  X  3  +  1  matrix  with  the  zero  digit 
centered  on  the  bottom  row. 

2.0  Alpha-numeric  keyboard. 

Keyboard  configurations  for  entry  of 
alphabetic  and  some  numeric 
information  shall  conform  to  MIL-STD- 
1280. 

3.0  Slope.  The  slope  of  nonportable 
keyboards  should  be  260-435  mrad  (15- 
25°)  from  the  horizontal.  The  preferred 
slope  is  280-300  mrad  (17-18°). 

4.0  Resistance.  The  force  required  to 
operate  alphanumeric  or  numeric 
keyboards  shall  conform  to  figure  11, 
page  91,  and  table  X,  page  95,  MIL- 
STD-1472D. 

Touch  Screen 

1.0  The  dimensions  and  separation  of 
responsive  areas  of  the  touch  screen 
shall  conform  to  SI,  S2,  and  S3  of 
figure  14,  page  119,  MIL-STD-1472D. 
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Evaluation 

Human  Factors  Principle 

Acceptable 

Acceptable  with 
Deficiencies 

Unacceptable 

Comments 

2.0  Resistance.  The  force  required  to 
operate  force-actuated  touch-screens 
shall  conform  to  the  alphanumeric 
resistance  limits  of  table  11,  page  91, 
MIL-STD-1472D. 

Equipment  Labeling 


1.0  Location.  The  gross  identifying 
label  on  a  unit,  assembly,  or  major 
subassembly  shall  be  located  externally, 
in  such  a  position  that  it  is  not  obscured 
by  adjacent  items;  on  the  flattest,  most 
uncluttered  surface  available;  and  on  a 
main  chassis  of  the  equipment. 

2.0  Functional  labeling.  Each  control 
and  display  shall  be  labeled  according  to 
function,  and  the  follovving  criteria  shall 
apply:  (a)  similar  names  for  different 
controls  and  displays  shall  be  avoided; 
(b)  instruments  shall  be  labeled  in  terms 
of  what  is  being  measured  or  controlled, 
taking  into  account  the  user  and 
purpose;  (c)  control  labeling  shall 
indicate  the  functional  result  of  control 
movement  (e.g.,  on,  off);  (d)  when 
controls  and  displays  must  be  used 
together  (in  certain  adjustment  tasks), 
appropriate  labels  shall  indicate  their 
functional  relationship  —  the  selection 
and  use  of  terminology  shall  be 
consistent. 


3.0  Location,  (a)  Ease  of  control 
op)eration  shall  be  given  priority  over 
visibility  of  labels;  (b)  labels  would 
normally  be  placed  above  the  controls 
and  displays  they  describe.  When  the 
panel  is  above  eye  level,  labels  may  be 
located  below  if  label  visibility  will  be 
enhanced. 


Workspace  Design 
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Evaluation 

Comments 

Acceptable 

Acceptable  with 
Deficiencies 

Unacceptable 

1.0  Kick  space.  All  cabinets,  consoles, 
and  work  surfaces  that  require  an 
operator  to  stand  or  sit  close  to  their 
front  surfaces  shall  contain  a  kick  space 
at  the  base  at  least  100  mm  (4  inches) 
deep  and  100  mm  (4  inches)  high. 

Human  Factors  Principle 

Evaluation 

Comments 

Acceptable 

Acceptable  with 
Deficiencies 

Unacceptable 

Standing  Operations 

1.0  Work  surface.  Work  surfaces 
designed  to  support  manual,  worksheets, 
etc,,  shall  be  915  +/-  mm  (36  +/-  0.6 
inches)  above  the  floor. 

2,0  Display  placement,  normal.  Visual 
displays  mounted  vertically  and  used  in 
normal  equipment  operation  shall  be 
placed  between  1,040  m  (41  inches)  and 
1.780  m  (70  inches)  above  the  standing 
surface. 

Seated  Operations 

1.0  Work  surface  width  and  depth.  A 
lateral  workspace  of  at  least  760  mm  (30 
inches)  wide  and  400  mm  (16  inches) 
deep  shall  be  provided  whenever 
practicable. 

2.0  Work  surface  height.  Desk  tops  and 
writing  tables  shall  be  740  to  790  mm 
(29  to  31  inches)  above  the  floor. 

Standard  Console  Design 
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Evaluation 

Comments 

Acceptable 

Acceptable  with 
Deficiencies 

Unacceptable 

1.0  Dimensions.  Consoles  which 
constitute  operator  workstations  should 
be  designed  to  conform  with  the 
dimensions  shown  in  table  XX  and 
figure  30,  pages  154-155,  MIL-STD- 
1472D. 

2.0  Viewing  angle.  The  total  required 
left-to-right  viewing  angle  shall  not 
exceed  190  degrees  (see  figure  2,  MIL- 
STD-1472D).  This  angle  should  be 
reduced  whenever  possible  through 
appropriate  control-display  layout. 
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1  Evaluation 

Comments 

Acceptable 

Acceptable  with 
Deficiencies 

Unacceptable 

6.0  Skills.  Access  to  items  maintained 
by  one  technician  should  not  require 
removal  of  critical  equipment 
maintained  by  another  technician, 
particularly  where  highly  specialized 
skills  are  involved. 

Covers 

1.0  Securing  covers.  It  shall  be  made 
obvious  when  a  cover  is  not  secured, 
even  though  it  may  be  in  place. 

2.0  If  the  method  of  opening  a  cover  is 
not  obvious  from  the  construction  of  the 
cover  itself,  instructions  shall  be 
permanently  displayed  on  the  outside  of 
the  cover.  Instructions  shall  consist  of 
simple  symbols  such  as  arrows  or  simple 
words  such  as  “push”  or  “pull.” 

3.0  Clearance.  Bulkheads,  brackets, 
and  other  equipment  shall  not  obstruct 
visual  or  physical  access  for  removal  or 
opening  of  covers  on  equipment  within 
which  work  must  be  performed  in  the 
installed  condition.  Covers,  doors,  or 
panels  which  must  be  opened  to  perform 
on-site  maintenance  shall  be  visually 
and  physically  accessible  to  the 
maintainers. 
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■  Not  at  all 

Very  much  so 

1  2  3 

_L_ 

4  r 

5 

1.  The  SPEARS  TIP  training  and  testing  improved 
my  ability  to  detect  explosive  devices. 

1 

2  3 

2.  The  SPEARS  TIP  training  and  testing  will  help 
me  on  my  job. 

1 

2  3 

4  ^ 

3.  I  enjoyed  taking  the  SPEARS  TIP  training  and 
testing. 

1 

2  3 

4  (5 

4.  I  would  like  to  continue  this  training  in  my 
work. 

1 

2  3 

0  5 

5.  The  SPEARS  TIP  function  is  easy  to  operate. 

1 

2  3 

4  (9 

6.  The  TIP  key  is  easy  to  locate  and  depress. 

1 

2  3 

7.  The  SPEARS  machine  showed  me  e.vplosive 
images  I  had  never  seen  before. 

1 

2  3 

4  (5  . 

S.  The  false  explosive  images  look  like  real  x-ray 
images  of  explosive  devices. 

1 

2  3 

^  0 

9.  I  could  not  guess  when  I  would  get  a  false 
explosive  image  from  the  SPEARS. 

1 

^  0 

4  5 

10.  The  SPEARS  generated  reports  were 
understandable. 

1 

2  3 

©  ’ 

1 1.  The  SPEARS  machine  helped  me  to 
understand  when  I  made  the  right  and  wrong 
decision  about  whether  there  was  an  explosive 
device  in  a  bag. 

1 

2  3 

0 , 

- .  12.  The  SPEARS  machine  did  not  slow  down  the 

movement  of  baggage  and  passengers. 

1 

“  © 

4  5 

13.  The  SPEARS  machine  did  not  interfere  with 
the  performance  of  my  tasks. 

1 

I  0 

4  5 

14.  The  SPEARS  machine  is  conveniently  located. 

1 

2  3 

0  ’ 

15.  The  SPEARS  machine  x-ray  enhancement 
buttons  improved  my  ability  to  detect  explosive 
devices. 

1 

2  3 

‘  0 
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ot  at  all 

Very  much  so 

1 

2 

J 

4 

5 

1.  The  SPEARS  TTP  training  and  testing  improved  1  2 

my  ability  to  detect  explosive  devices. 

2.  The  SPEARS  TIP  training  and  testing  will  help  1  2 

me  on  my  job. 

3.  I  enjoyed  taking  the  SPEARS  TIP  training  and  1  2 

testing. 

4.  I  would  like  to  continue  this  training  in  my  12 

work. 

5.  The  SPEARS  TIP  function  is  easy  to  operate.  1  2 

6.  The  TIP  key  is  easy  to  locate  and  depress.  ,1  2 

7.  The  SPEARS  machine  showed  me  e.\plosive  1  2 

images  I  had  never  seen  before. 

w 

S.  The  false  explosive  images  look  like  real  .x-ray  1  2 

images  of  explosive  devices. 

9.  I  could  not  guess  when  I  would  get  a  false  1 

explosive  image  from  the  SPEARS. 

10.  The  SPEARS  generated  reports  were  1  2 

understandable. 

11.  The  SPEARS  machine  helped  me  to  12 


understand  when  I  made  the  right  and  wrong 
decision  about  whether  there  was  an  explosive 
device  in  a  bag. 

12.  The  SPEARS  machine  did  not  slow  down  the  12 
movement  of  baggage  and  passengers. 

t 

13.  The  SPEARS  machine  did  not  interfere  with  1  2 

the  performance  of  my  tasks. 

14.  The  SPEARS  machine  is  conveniently  located.  1  2 

15.  The  SPEARS  machine  x-ray  enhancement  1  2 

buttons  improved  my  ability  to  detect  explosive 

devices. 


d)  4  5 

3  0  5 

3  Q  5 
3  4  Q) 

3  4 

3  4  Q 

3  (9  5 

3  Q  5 
3  4  5 

3  4  5 

3  0)  5. 

3  Q  5 

3  4  (p 

3  4  (f) 
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‘  Not  at  all 


1 


1.  The  SPEARS  TIP  training  and  testing  improved  1 
my  ability  to  detect  explosive  devices. 

2.  The  SPEARS  TIP  training  and  testing  will  help  1 
me  on  my  job. 

3.  I  enjoyed  taking  the  SPEARS  TIP  training  and  1 
testing. 

4.  I  would  like  to  continue  this  training  in  my  1 

work. 


Very  much  so 


(£)  4  5 


(i)  '•  5 

3  ^  5 


5.  The  SPEARS  TIP  function  is  easy  to  operate. 

6.  The  TIP  key  is  easy  to  locate  and  depress. 

7.  The  SPEARS  machine  showed  me  e.\piosivc 
images  I  had  never  seen  before. 

S.  The  false  explosive  images  look  like  real  x-ray 
images  of  explosive  devices. 

9.  I  could  not  guess  when  I  would  get  a  false 
explosive  image  from  the  SPEARS. 

10.  The  SPEARS  generated  reports  were 
understandable. 


12  3  4 


12  3  4 


1^3  4  5 

1  2  (3)  4  3 


1  2 


4  5 


1  2  3  (4  5 


11.  The  SPEARS  machine  helped  me  to 
understand  when  I  made  the  right  and  wrong 
decision  about  whether  there  was  an  explosive 


device  in  a  bag, 
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1  2 


4  5 


12.  The  SPEARS  machine  did  not  slow  down  the 
movement  of  baggage  and  passengers. 

13.  The  SPEARS  machine  did  not  interfere  with 
the  performance  of  my  tasks. 


14.  The  SPEARS  machine  is  conveniently  located.  1  2  3  4  (5 


3  4  5 


1  2  3  4  5 


15.  The  SPEARS  machine  .\-ray  enhancement 
buttons  improved  my  ability  to  detect  explosive 
devices. 
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lot  at  all 

Very  much  so 

1 

2 
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1.  The  SPEARS  TIP  training  and  testing  improved  1 
my  ability  to  detect  explosive  devices. 

2.  The  SPEARS  TIP  training  and  testing  will  help  1 
me  on  my  job. 

3.  I  enjoyed  taking  the  SPEARS  TIP  training  and  1 
testing. 

4.  I  would  like  to  continue  this  training  in  my  1 

work. 


2  3  4  (J* 

2  3  4  (9 

2  3  4 

2  3  4 


5.  The  SPEARS  TIP  function  is  easy  to  operate.  12  3 

6.  The  TIP  key  is  easy  to  locate  and  depress.  .1  2  3 

7.  The  SPEARS  machine  showed  me  e.\plosive  1  2  3 

images  I  had  never  seen  before. 

S.  The  false  e.xplosive  images  look  like  real  x-ray  12  3 

images  of  explosive  devices. 

9.  I  could  not  guess  when  I  would  get  a  false  1  2  3 

explosive  image  from  the  SPEARS. 


10.  The  SPEARS  generated  reports  were  12  3 

understandable. 


11.  The  SPEARS  machine  helped  me  to  12  3 

understand  when  I  made  the  right  and  wrong 

decision  about  whether  there  was  an  explosive 
device  in  a  bag. 

12.  The  SPEARS  machine  did  not  slow  down  the  12  3 

movement  of  baggage  and  passengers. 

\ 

13.  The  SPEARS  machine  did  not  interfere  with  12  3 

the  performance  of  my  tasks. 


14.  The  SPEARS  machine  is  conveniently  located.  12  3 
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15.  The  SPEARS  machine  x-ray  enhancement  1  2  3  4  Cp 

buttons  improved  my  ability  to  detect  explosive 

devices. 
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Very  much  so 


ot  at  all 
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1.  The  SPEARS  TIP  training  and  testing  improved 
my  ability  to  detect  explosive  devices. 


2.  The  SPEARS  TIP  training  and  testing  will  help  1 
me  on  my  job. 


3.  I  enjoyed  taking  the  SPEARS  TIP  training  and  1 
testing. 

4.  I  would  like  to  continue  this  training  in  my  1 

work. 

5.  The  SPEARS  TIP  function  is  easy  to  operate.  1 


6.  The  TIP  key  is  easy  to  locate  and  depress.  1 


7.  The  SPEARS  machine  showed  me  e.\*plosive  1 
images  I  had  never  seen  before. 

S.  The  false  e.xplosive  images  look  like  real  x-ray  1 
images  of  explosive  devices. 

9.  I  could  not  guess  when  I  would  get  a  false 
explosive  image  from  the  SPEARS. 

10.  The  SPEARS  generated  reports  were 
understandable. 

11.  The  SPEARS  machine  helped  me  to 
understand  when  I  made  the  right  and  wrong 
decision  about  whether  there  was  an  explosive 
device  in  a  bag. 

12.  The  SPEARS  machine  did  not  slow  down  the  1 
movement  of  baggage  and  passengers. 

13.  The  SPEARS  machine  did  not  interfere  with  1 
the  performance  of  my  tasks. 

14.  The  SPEARS  machine  is  conveniently  located.  1 

15.  The  SPEARS  machine  x-ray  enhancement  1 

buttons  improved  my  ability  to  detect  e.xplosive 
devices. 
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Not  at  all 
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Very  much  so 
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1.  The  SPEARS  TIP  training  and  testing  improved 
my  ability  to  detect  explosive  devices. 


4 


5 


2.  The  SPEARS  TIP  training  and  testing  will  help 
me  on  my  job. 


4 


5 


3.  I  enjoyed  taking  the  SPEARS  TIP  training  and 
testing. 


7 


j 


4 
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4.  I  would  like  to  continue  this  training  in  my 
work. 


5 


5.  The  SPEARS  TIP  function  is  easy  to  operate. 


2  3 
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6.  The  TIP  key  is  easy  to  locate  and  depress. 

7.  The  SPEARS  machine  showed  me  e.\*plosive 
images  I  had  never  seen  before. 


4  5 

4  5 


S.  The  false  e.xplosive  images  look  like  real  x-ray 
images  of  explosive  devices. 


2  3 
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9.  I  could  not  guess  when  I  would  get  a  false 
explosive  image  from  the  SPEARS. 

10.  The  SPEARS  generated  reports  were 
understandable. 
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1  2  3  '4  0 


11.  The  SPEARS  machine  helped  me  to 
understand  when  I  made  the  right  and  wrong 
decision  about  whether  there  was  an  explosive 
device  in  a  bag. 


12.  The  SPEARS  machine  did  not  slow  down  the  12 
movement  of  baggage  and  passengers. 

13.  The  SPEARS  machine  did  not  interfere  with  1  2 

the  performance  of  my  tasks. 


14.  The  SPEARS  machine  is  conveniently  located. 

15.  The  SPEARS  machine  x-ray  enhancement 
buttons  improved  my  ability  to  detect  explosive 
devices. 
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Very  much  so 


Not  at  all 
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1.  The  SPEARS  TIP  training  and  testing  improved  1 
my  ability  to  detect  explosive  devices. 

2.  The  SPEARS  TIP  training  and  testing  will  help  1 
me  on  my  job. 


3.  I  enjoyed  taking  the  SPEARS  TIP  training  and  1 
testing. 

4.  I  would  like  to  continue  this  training  in  my  1 

work. 


3  CP  5 

3  5 


5.  The  SPEARS  TIP  function  is  easy  to  operate. 


4 


6.  The  TIP  key  is  easy  to  locate  and  depress. 


2  3  4 


7.  The  SPEARS  machine  showed  me  e.xplosive  12  3  ^  5 

images  I  had  never  seen  before. 

w 


S.  The  false  explosive  images  look  like  real  x-ray  1 
images  of  explosive  devices. 

9.  I  could  not  guess  when  I  would  get  a  false  1 

explosive  image  from  the  SPEARS. 
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10.  The  SPEARS  generated  reports  were 
understandable. 
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11.  The  SPEARS  machine  helped  me  to 
understand  when  I  made  the  right  and  wrong 
decision  about  whether  there  was  an  explosive 
device  in  a  bag. 


12.  The  SPEARS  machine  did  not  slow  down  the 
movement  of  baggage  and  passengers. 

t 

13.  The  SPEARS  machine  did  not  interfere  with 
the  performance  of  my  tasks. 


1 


14.  The  SPEARS  machine  is  conveniently  located.  1 


15.  The  SPEARS  machine  x-ray  enhancement  1 

buttons  improved  my  ability  to  detect  explosive 
devices. 
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Not  at  all  _ _ 

I  1  I  2  I  3  I 

1.  The  SPEARS  TIP  training  and  testing  improved  1 
my  ability  to  detect  explosive  devices. 

2.  The  SPEARS  TIP  training  and  testing  will  help  1 
me  on  my  job. 

3.  I  enjoyed  taking  the  SPEARS  TIP  training  and  1 
testing. 

4.  I  would  like  to  continue  this  training  in  my  1 

work. 

5.  The  SPEARS  TIP  function  is  easy  to  operate.  1  • 

6.  The  TIP  key  is  easy  to  locate  and  depress.  1 


7.  The  SPEARS  machine  showed  me  explosive 
images  I  had  never  seen  before. 

S.  The  false  explosive  images  look  like  real  .x-ray 
images  of  explosive  devices. 

9.  I  could  not  guess  when  I  would  get  a  false 
explosive  image  from  the  SPEARS. 

10.  The  SPEARS  generated  reports  were 
understandable. 

11.  The  SPEARS  machine  helped  me  to 
understand  when  I  made  the  right  and  wrong 
decision  about  whether  there  was  an  explosive 
device  in  a  bag. 

12.  The  SPEARS  machine  did  not  slow  down  the 
movement  of  baggage  and  passengers. 

\ 

13.  The  SPEARS  machine  did  not  interfere  with 
the  performance  of  my  tasks. 

14.  The  SPEARS  machine  is  conveniently  located.  1 

15.  The  SPEARS  machine  ,x-ray  enhancement  1 

buttons  improved  my  ability  to  detect  explosive 
devices. 


1 

1 

1 

1 

1 


_ Very  much  so 

4  I  5 


5 


5 


4 


5 


2 


4 


2 


4  ■ 


2  3  4 


C-9 


Not  at  all 
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1.  The  SPEARS  TIP  training  and  testing  improved 
my  ability  to  detect  explosive  devices, 

2.  The  SPEARS  TIP  training  and  testing  will  help 
me  on  my  job. 

3.  I  enjoyed  taking  the  SPEARS  TIP  training  and 
testing. 

4.  I  would  like  to  continue  this  training  in  my 
work. 

5.  The  SPEARS  TIP  function  is  easy  to  operate. 

6.  The  TIP  key  is  easy  to  locate  and  depress. 

7.  The  SPEARS  machine  showed  me  e.xplosive 
images  I  had  never  seen  before. 

S.  The  false  explosive  images  look  like  real  x-ray 
images  of  explosive  devices. 

9.  I  could  not  guess  when  I  would  get  a  false 
explosive  image  from  the  SPEARS. 

10.  The  SPEARS  generated  reports  were 
understandable. 

11.  The  SPEARS  machine  helped  me  to 
understand  when  I  made  the  right  and  wrong 
decision  about  whether  there  was  an  explosive 
device  in  a  bag. 

12.  The  SPEARS  machine  did  not  slow  down  the 
movement  of  baggage  and  passengers. 

\ 

13.  The  SPEARS  machine  did  not  interfere  with 
the  performance  of  my  tasks. 

14.  The  SPEARS  machine  is  conveniently  located. 

15.  The  SPEARS  machine  x-ray  enhancement 
buttons  improved  my  ability  to  detect  explosive 
devices. 
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Not  at  all  _ _ 
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1.  The  SPEARS  TIP  training  and  testing  improved 
my  ability  to  detect  explosive  devices. 

2.  The  SPEARS  TIP  training  and  testing  will  help 
me  on  my  job. 

3.  I  enjoyed  taking  the  SPEARS  TIP  training  and 
testing. 

4.  I  would  like  to  continue  this  training  in  my 
work. 

5.  The  SPEARS  TIP  function  is  easy  to  operate. 

6.  The  TIP  key  is  easy  to  locate  and  depress. 

7.  The  SPEARS  machine  showed  me  explosive 
images  I  had  never  seen  before. 

S.  The  false  explosive  images  look  like  real  x-ray 
images  of  explosive  devices. 

9.  I  could  not  guess  when  1  would  get  a  false 
explosive  image  from  the  SPEARS. 

10.  The  SPEARS  generated  reports  were 
understandable. 

11.  The  SPEARS  machine  helped  me  to 
understand  when  I  made  the  right  and  wrong 
decision  about  whether  there  was  an  explosive 
device  in  a  bag. 

12.  The  SPEARS  machine  did  not  slow  down  the 
movement  of  baggage  and  passengers. 

t 

13.  The  SPEARS  machine  did  not  interfere  with 
the  performance  of  my  tasks. 

14.  The  SPEARS  machine  is  conveniently  located. 

15.  The  SPEARS  machine  .x-ray  enhancement 
buttons  improved  my  ability  to  detect  explosive 
devices. 
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Not  at  all 
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Very  much  so 
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1.  The  SPEARS  TIP  training  and  testing  improved  1 
my  ability  to  detect  explosive  devices. 

2.  The  SPEARS  TIP  training  and  testing  will  help  1 
me  on  my  job. 

3.  I  enjoyed  taking  the  SPEARS  TIP  training  and  1 
testing. 

4.  I  would  like  to  continue  this  training  in  my  1 

work. 

5.  The  SPEARS  TIP  function  is  easy  to  operate.  1 

6.  The  TIP  key  is  easy  to  locate  and  depress.  1 

7.  The  SPEARS  machine  showed  me  e.xplosive  1 

images  I  had  never  seen  before. 

S.  The  false  e.\plosive  images  look  tike  real  .x-ray  1 
images  of  explosive  devices. 

9.  I  could  not  guess  when  I  would  get  a  false  ^  1 

explosive  image  from  the  SPEARS. 

10.  The  SPEARS  generated  reports  were  1 

understandable. 

11.  The  SPEARS  machine  helped  me  to  1 


understand  when  I  made  the  right  and  wrong 
decision  about  whether  there  was  an  e.xplosive 
device  in  a  bag. 

12.  The  SPEARS  machine  did  not  slow  down  the 
movement  of  baggage  and  passengers. 

t 

13.  The  SPEARS  machine  did  not  interfere  with 
the  performance  of  my  tasks. 

14.  The  SPEARS  machine  is  conveniently  located.  1 

15.  The  SPEARS  machine  x-ray  enhancement  1 

buttons  improved  my  ability  to  detect  explosive 
devices. 
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1,  The  SPEARS  TIP  training  and  testing  improved  1 
my  ability  to  detect  explosive  devices. 

2.  The  SPEARS  TIP  training  and  testing  will  help  1 
me  on  my  job. 


3.  I  enjoyed  taking  the  SPEARS  TIP  training  and 
testing. 


4.  I  would  like  to  continue  this  training  in  my 
work. 


5.  The  SPEARS  TIP  function  is  easy  to  operate. 


6.  The  TIP  key  is  easy  to  locate  and  depress. 


7.  The  SPEARS  machine  showed  me  explosive  1  2  3  4  5 

images  I  had  never  seen  before. 

S.  The  false  explosive  images  look  like  real  x-ray  12  3  (4^  5 

images  of  explosive  devices. 


9.  I  could  not  guess  when  1  would  get  a  false  1  2 

explosive  image  from  the  SPEARS. 

10.  The  SPEARS  generated  reports  were  1  2 

understandable. 

11.  The  SPEARS  machine  helped  me  to  1  2 

understand  when  I  made  the  right  and  wrong 

decision  about  whether  there  was  an  explosive 
device  in  a  bag. 

12.  The  SPEARS  machine  did  not  slow  down  the  12 
movement  of  baggage  and  passengers. 

t 

13.  The  SPEARS  machine  did  not  interfere  with  1  2 

the  performance  of  my  tasks. 

14.  The  SPEARS  machine  is  conveniently  located.  1  2 

15.  The  SPEARS  machine  x-ray  enhancement  1  2 

buttons  improved  my  ability  to  detect  e.xplosive 

devices. 
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1.  The  SPEARS  TIP  training  and  testing  improved  12  3 

my  ability  to  detect  explosive  devices. 


2.  The  SPEARS  TIP  training  and  testing  will  help  12  3  4 

me  on  my  job. 

3.  I  enjoyed  taking  the  SPEARS  TIP  training  and  12  3  4 

testing. 

4.  I  would  like  to  continue  this  training  in  my  12  3  4 

work. 

5.  The  SPEARS  TIP  function  is  easy  to  operate.  1  2  3  4 

6.  The  TIP  key  is  easy  to  locate  and  depress.  ,1  2  3  4 

7.  The  SPEARS  machine  showed  me  e.\plosive  12  3  4 

images  I  had  never  seen  before. 

S.  The  false  explosive  images  look  like  real  x-ray  12  3  4 

images  of  explosive  devices. 

9.  I  could  not  guess  when  I  would  get  a  false  12  3  4 

explosive  image  from  the  SPEARS. 


10.  The  SPEARS  generated  reports  were  1  2  3  4  .5/ 

understandable.  i 


11.  The  SPEARS  machine  helped  me  to  1  2 

understand  when  I  made  the  right  and  wrong 

decision  about  whether  there  was  an  explosive 
device  in  a  bag. 

12.  The  SPEARS  machine  did  not  slow  down  the  12 
movement  of  baggage  and  passengers. 

13.  The  SPEARS  machine  did  not  interfere  with  1  2 

the  performance  of  my  tasks. 

14.  The  SPEARS  machine  is  conveniently  located.  1  2 

15.  The  SPEARS  machine  .x-ray  enhancement  1  2 

buttons  improved  my  ability  to  detect  explosive 

devices. 
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Not  at  all 
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1.  The  SPEARS  TIP  training  and  testing  improved  1 
my  ability  to  detect  explosive  devices. 


2.  The  SPEARS  TIP  training  and  testing  will  help  1 
me  on  my  job. 


3.  I  enjoyed  taking  the  SPEARS  TIP  training  and 
testing. 

4.  I  would  like  to  continue  this  training  in  my 
work. 


5.  The  SPEARS  TIP  function  is  easy  to  operate.  1 


6.  The  TIP  key  is  easy  to  locate  and  depress. 


7.  The  SPEARS  machine  showed  me  e.xplosive 
images  I  had  never  seen  before. 


S.  The  false  e.xplosive  images  look  like  real  .x-ray  1 
images  of  explosive  devices. 

9.  I  could  not  guess  when  I  would  get  a  false  1 

explosive  image  from  the  SPEARS. 

10.  The  SPEARS  generated  reports  were  1 

understandable. 


11.  The  SPEARS  machine  helped  me  to  1 

understand  when  I  made  the  right  and  wrong 
decision  about  whether  there  was  an  explosive 
device  in  a  bag. 


12.  The  SPEARS  machine  did  not  slow  down  the  1 
movement  of  baggage  and  passengers. 


13.  The  SPEARS  machine  did  not  interfere  with 
the  performance  of  my  tasks. 


14.  The  SPEARS  machine  is  conveniently  located. 

15.  The  SPEARS  machine  x-ray  enhancement 
buttons  improved  my  ability  to  detect  e.xplosive 
devices. 
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1.  The  SPEARS  TIP  training  and  testing  improved 
my  ability  to  detect  explosive  devices. 

2.  The  SPEARS  TIP  training  and  testing  will  help 
me  on  my  job. 

3.  I  enjoyed  taking  the  SPEARS  TIP  training  and 
testing. 

4.  I  would  like  to  continue  this  training  in  my 
work. 

5.  The  SPEARS  TIP  function  is  easy  to  operate. 

6.  The  TIP  key  is  easy  to  locate  and  depress. 

7.  The  SPEARS  machine  showed  me  e.xplosive 
images  I  had  never  seen  before. 

S.  The  false  e.xplosive  images  look  like  real  x-ray 
images  of  explosive  devices. 

9.  I  could  not  guess  when  I  would  get  a  false 
explosive  image  from  the  SPEARS. 

10.  The  SPEARS  generated  reports  were 
understandable. 

11.  The  SPEARS  machine  helped  me  to 
understand  when  I  made  the  right  and  wrong 
decision  about  whether  there  was  an  explosive 
device  in  a  bag. 

12.  The  SPEARS  machine  did  not  slow  down  the 
movement  of  baggage  and  passengers. 

\ 

13.  The  SPEARS  machine  did  not  interfere  with 
the  performance  of  my  tasks. 

14.  The  SPEARS  machine  is  conveniently  located. 


1 


1 


1 


15.  The  SPEARS  machine  x-ray  enhancement  1 

buttons  improved  my  ability  to  detect  explosive 
devices. 


Not  at  all  _ _ 

1  I  2  I  3  I  4 


1.  The  SPEARS  TIP  training  and  testing  improved  1 . 
my  ability  to  detect  explosive  devices. 


2.  The  SPEARS  TIP  training  and  testing  will  help 
me  on  my  job. 

3.  I  enjoyed  taking  the  SPEARS  TIP  training  and 
testing. 

4.  I  would  like  to  continue  this  training  in  my 
work. 


0 


7 


7 


5.  The  SPEARS  TIP  function  is  easy  to  operate.  1 


Very  much  so 


© 


j 


5 


5 


6.  The  TIP  key  is  easy  to  locate  and  depress. 

7.  The  SPEARS  machine  showed  me  explosive 
images  I  had  never  seen  before. 

S.  The  false  explosive  images  look  like  real  x-ray 
images  of  explosive  devices. 

9.  I  could  not  guess  when  1  would  get  a  false 
explosive  image  from  the  SPEARS. 

10.  The  SPEARS  generated  reports  were 
understandable. 


1 1.  The  SPEARS  machine  helped  me  to 
understand  when  I  made  the  right  and  wrong 
decision  about  whether  there  was  an  e.xplosive 
device  in  a  bag. 

12.  The  SPEARS  machine  did  not  slow  down  the 
movement  of  baggage  and  passengers. 


1 


2 


2 


13.  The  SPEARS  machine  did  not  interfere  with  1 
the  performance  of  my  tasks. 


14.  The  SPEARS  machine  is  conveniently  located. 


j 
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4 

4 
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5 


15,  The  SPEARS  machine  .x-ray  enhancement 
buttons  improved  my  ability  to  detect  explosive 
devices. 


7 


4  5 
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1 


Not  at  all 


1 


1.  The  SPEARS  TIP  training  and  testing  improved 
my  ability  to  detect  explosive  devices. 

2.  The  SPEARS  TIP  training  and  testing  will  help 
me  on  my  job. 

3.  I  enjoyed  taking  the  SPEARS  TIP  training  and 
testing. 

4.  I  would  like  to  continue  this  training  in  my 
work. 

5.  The  SPEARS  TIP  function  is  easy  to  operate. 

6.  The  TIP  key  is  easy  to  locate  and  depress. 

V 

7.  The  SPEARS  machine  showed  me  e.\plosive 
images  I  had  never  seen  before. 

S.  The  false  e.xplosive  images  look  like  real  .x-ray 
images  of  explosive  devices. 

9.  I  could  not  guess  when  I  would  get  a  false 
explosive  image  from  the  SPEARS. 

10.  The  SPEARS  generated  reports  were 
understandable. 

11.  The  SPEARS  machine  helped  me  to 
understand  when  I  made  the  right  and  wrong 
decision  about  whether  there  was  an  e.xplosive 
device  in  a  bag. 

12.  The  SPEARS  machine  did  not  slow  down  the 
movement  of  baggage  and  passengers. 

\ 

13.  The  SPEARS  machine  did  not  interfere  with 
the  performance  of  my  tasks. 

14.  The  SPEARS  machine  is  conveniently  located. 

15.  The  SPEARS  machine  .x-ray  enhancement 

buttons  improved  my  ability  to  detect  e.xplosive 
devices.  ■ 


1 

1 

1 

1 

1 

1 

1 

1 

1 

1 


1 

1 

1 
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SPEARS  IMAGE  CONTENT  CHECKLIST 
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SPEARS  IMAGE  CONTENT  CHECKLIST 


Evaluation 


SPEARS  Image  Content 

Yes 

1.  The  number  of  images  of 
CTIs  provided  by  the  system  is 

NA 

2.  The  number  of  images  of 
FTIs  provided  by  the  system  is 

NA 

3.  The  number  of  CTIs  of 
weapons  is 

NA 

4.  The  number  of  CTIs  of 
explosives  is 

NA 

5.  The  number  of  CTIs  of 
incendiary  devices  is 

NA 

6.  The  number  of  CTIs  of 
dangerous/deadly  devices  other 
than  weapons,  incendiaries  or 
explosives  is 

NA 

7.  The  number  of  FTIs  of 
weapons  is 

NA 

8.  The  number  of  FTIs  of 
explosives  is 

NA 

9.  The  number  of  FTIs  of 
dangerous/deadly  devices  other 
than  weapons,  incendiaries  or 
explosives  is 

NA 

10.  At  least  two  different 
aspect  angles  are  available  for 
each  CTI  presentation. 

NA 

11.  At  least  two  different 
aspect  angles  are  available  for 
each  FTI  presentation. 

X 

No _ Comments/Description 

NA 

NA 

NA 

NA 

NA 

NA 

NA  89 +7  FAA  test  obj.  =  96 

NA  Explosives  =  65 
Components  =  55 

NA 

NA 
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SPEARS  CUSTOMIZATION  CHECKLIST 
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SPEARS  CUSTOMIZATION  CHECKLIST 


Evaluation 

SPEARS  Customization 

Yes 

No 

1.  Combined  Threats  and  Images 
(CTIs)  can  be  selected  for  each 
screener  as  a  function  of  threat 
category. 

X 

2.  CTIs  can  be  selected  for  each 
screener  as  a  function  of  difficulty. 

X 

3.  CTIs  can  be  selected  for  each 
screener  as  a  function  of  aspect 
angle. 

X 

4.  CTIs  can  be  selected  as  a 
function  of  the  combination  of  threat 
category,  screener  difficulty,  and 
aspect  angle. 

X 

5.  CTIs  can  be  archived  with  the 
associated  screener  outcome  ( hit, 
miss,  false  alarm,  and  correct 
rejection). 

X 

6.  Each  archived  CTI  with 
associated  screener  trial  outcome 
includes  date  and  time  of  day. 

X 

7.  Fictional  Threat  Images  (FTIs) 
can  be  selected  for  each  screener  as 
a  function  of  threat  category. 

X 

8.  FTIs  can  be  selected  for  each 
screener  as  a  function  of  difficulty. 

X 

9.  FTIs  can  be  selected  for  each 
screener  as  a  function  of  aspect 
angle. 

X 

Evaluation 


SPEARS  Customization 

Yes 

No 

10.  FTIs  can  be  selected  as  a 
function  of  the  combination  of  threat 
category,  screener  difficulty,  and 
aspect  angle. 

X 

1 1 .  FTIs  can  be  archived  with  the 
associated  screener  outcome  (hit, 
miss,  false  alarm,  correct  rejection). 

X 

12.  Each  archived  FTI  with 
associated  screener  trial  outcome 
includes  date  and  time  of  day. 

X 

13.  CTIs  can  be  automatically  or 
manually  selected  based  on  time  of 
day. 

X 

14.  CTIs  can  be  automatically  or 
manually  selected  based  on 
checkpoint  activity. 

X 

15.  CTIs  can  be  automatically  or 
manually  selected  based  on  screener 
identity. 

X 

16.  CTIs  can  be  automatically  or 
manually  selected  based  on  the 
combination  of  time  of  day, 
checkpoint  activity,  and  screener 
identity. 

X 

17.  FTIs  can  be  automatically  or 
manually  selected  based  on  time  of 
day. 

X 

18.  FTIs  can  be  automatically  or 
manually  selected  based  on 
checkpoint  activity. 

X 

19.  FTIs  can  be  automatically  or 
manually  selected  based  on  screener 
identity. 

X 
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SPEARS  FEEDBACK  CHECKLIST 
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SPEARS  FEEDBACK  CHECKLIST 


Evaluation 

SPEARS  Feedback 

Yes 

No 

Comments/Description 

1 .  Feedback  is  immediate  after 
correct,  incorrect,  or  missed 
identification  of  an  FTI. 

X 

Slow  -  Missed  threats  are  not 
identified  and/or  disappear  too 
quickly 

2.  Feedback  is  displayed  in  a 
consistent  position. 

X 

3.  Feedback  is  consistent  in  terms 
of  format  and  content. 

X 

4.  Feedback  is  provided  for  correct 
FTI  identification. 

X 

5.  Feedback  is  provided  for  missed 
FTI  identification. 

X 

The  actual  FTI  disappears  too 
quickly 

6.  Following  missed  FTI 
identification,  the  location  of  the 
missed  FTI  is  provided  on  the 
display. 

X 

7.  Feedback  is  provided  for  threat 
items  that  do  not  contain  threat 
images. 

X 

8.  FTIs  are  removed  from  the 
display  to  allow  additional  image 
scanning. 

X 

Operators  indicate  image  removal  is 
too  slow 

9.  The  time  interval  between  user 
initiation  and  system  feedback  for 
each  condition  of  correct  or  missed 
identification,  image  removal,  and 
threat  items  which  do  not  contain 
threat  images  is 

NA 

NA 
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Evaluation 

SPEARS  Feedback 

Yes 

No 

Comments/Description 

10.  The  nature  of  the  feedback  (i  .e., 
visual  and/or  auditory,  graphical, 
textual,  etc.)  for  each  condition  of 
correct  or  missed  identification, 
image  removal,  and  threat  items 
which  do  not  contain  threat  images 
is 

NA 

NA 

textual 
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SPEARS  CAPABILITIES  SUMMARIES  CHECKLIST 
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SPEARS  REPORT  SUMMARIES  CHECKLIST 


SPEARS  Reports 


1.  Summary  reports  are 
automatically  generated. 

2.  Summary  reports  are  generated 
on  demand. 

3.  Summary  reports  contain 
descriptive  statistics. 

4.  Descriptive  statistics  are 
categorized  according  to; 

a.  threats  presented  and 
outcomes 

b.  screener  identification 

c.  checkpoint  activity  level 

d.  time  of  day 

5.  Summary  reports  are  readily 
understandable  by  the  user  (based  on 
structured  interview  question 
results). 

6.  Performance  levels  for  individual 
screeners  per  screening  session  are 
actively  and  accurately  reported  as  a 
function  of  the  probability  of 
detection  (Pd). 

I  7.  Performance  level  reports  can  be 
provided  to  a  remotely  situated 
supervisor. 

8.  A  remotely  situated  supervisor 
can  be  notified  when  performance 
levels  are  above  or  below  supervisor 
preset  performance  criteria 
thresholds. 


Evaluation 


Yes  I  No  I  Comments/Description 
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INTEROPERABILITY  CHECKLIST 
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INTEROPERABILITY  CHECKLIST 


SPEARS  Interoperabilit 


1 .  A  connection  via  modem  can  be 
established  between  two  Screener 
Proficiency  Evaluation  and 
Reporting  System  (SPEARS) 
devices  or  a  SPEARS  device  and  a 
computer  over  standard  telephone 
lines. 

2.  The  SPEARS  remote  connection 
requires  no  more  than  two  people 
(one  on-site,  one  at  the  remote  site) 
to  operate. 

3.  New  CTIs  can  be  downloaded 
remotely  to  the  SPEARS  on-site 
device. 

4.  New  FTIs  can  be  downloaded 
to  the  SPEARS  on-site  device. 

5.  Status  reports  can  be  initiated 
remotely  via  the  modem  connection. 

6.  Status  report  data  can  be 
accessed  remotely  and  saved  to  a 
local  computer  file. 

7.  Status  reports  can  be  printed  out 
from  the  remote  location. 

8.  Remotely  accessed  status  reports 
are  similar  to  those  available  on-site 
from  the  SPEARS  device. 

9.  The  time  required  to  transfer  a 

CTI  remotely  is _ . 

10.  The  time  required  to  transfer  an 

FTI  remotely  is _ . 


Evaluation 


Yes  No 


Comments/Description 
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SECURITY  ACCESS  CONTROL  CHECKLIST  -  TnT 
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SECURITY  ACCESS  CONTROL  CHECKT.TST 


Evaluation 

Security  Access  Control 

Yes 

No 

Comments/Description 

1  The  system  provides  controls  to  limit 
access  rights. 

X 

2.  Access  control  can  be  limited  to  the 
granularity  of  a  single  user. 

X 

3.  The  system  provides  controls  to  limit 
propagation  of  sensitive  information. 

X 

4.  Files  are  protected  from  unauthorized 
access. 

X 

5.  Files  are  sensitivity  labeled. 

X 

The  TnT  does  not  use  sensitivity 
labels.  Files  are  not  accessible  to 
individuals  who  are  not  authorized. 

6.  File  sensitivity  labels  are  used  as  a 
basis  for  access  control  decisions. 

X 

No  sensitivity  labels.  Individuals  are 
prevented  from  accessing  files  if  not 
authorized. 

7.  Data  are  protected  during 
transmission. 

X 

8.  When  data  are  exported,  a  sensitivity 
label  is  included  that  accurately 
represents  the  level  of  security  of  the 
file. 

X 

No  sensitivity  labels.  Files  are 
accessible  to  individuals  who  are  not 
authorized.  The  only  data  that  can  be 
exported  are  system  back-ups,  which 
are  put  on  a  floppy  disk.  The 
information  is  encoded  in  an  EG&G 
propriety  format  that  only  they  can 
decode.  Operating  system  is  password 
protected. 

9.  Procedures  are  established  to  control 
access  to  printed  data. 

X 

10.  Users  can  access  files  only  if  their 
security  level  is  greater  than  or  equal  to 
the  security  level  of  the  file. 

X 
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Security  Access  Control 


1 1 .  Users  are  required  to  enter 
identification  before  performing  any 
actions. 


Evaluation 


Yes  I  No 


X 


Comments/Description 


12.  The  system  maintains  data  to 
authenticate  user  identity  and  level  of 
access. 

13.  Authentication  data  cannot  be 
accessed  by  any  unauthorized  user. 

14.  The  system  maintains  an  access 
audit  trail. 

15.  The  access  audit  trail  is  protected 
from  modification,  unauthorized  access, 
or  destruction. 

16.  The  security  mechanism  of  the 
system  can  be  tested  and  works  as 
claimed  in  the  system  documentation. 

17.  Anyone  external  to  those  who  have 
been  granted  access  privileges  cannot 
read,  change,  or  delete  data. 

18.  Documentation  is  provided  that 
describes  the  protection  mechanisms 
provided  by  the  system,  and  their  use. 


19.  A  manual  is  provided  to  the  system 
administrator  that  presents  cautions 
about  functions  and  privileges  that 
should  be  controlled  to  maintain  a 
secure  system. 

20.  The  manual  describes  the  operator 
and  administrator  functions  related  to 
security. 
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SPEARS  IMAGE  INSERTION  ISSUE  CHECKLIST 
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SPEARS  IMAGE  INSERTION  ISSUE  CHECKLIST 


Evaluation 

SPEARS  Insertion 

Yes 

No 

Comments/Description 

1.  FTIs  can  be  generated  using  a 
computer  generated  threat  object 
and  neutral  baggage  image. 

X 

2.  FTIs  can  be  automatically 

X 

FTIs  cannot  be  generated  by 

generated. 

category 

3.  FTIs  are  not  predictable  in  the 
order  they  are  presented. 

X 

4.  CTIs  can  be  inserted  into  the 
normal  baggage  flow. 

X 

5.  CTIs  can  be  automatically 
generated. 

X 

6.  CTIs  are  not  predictable  in  the 
order  they  are  presented. 

X 

7.  When  an  FTI  and  neutral  bag 
image  are  combined,  FTIs 
perceptually  appear  inside  the 
neutral  bag. 

8.  FTI  position  in  the  neutral  bag 
can  be  randomly  determined. 

9.  FTI  orientation  in  the  neutral 
bag  can  be  controlled. 
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SUBJECT  INFORMED  CONSENT  FORM 
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INFORMED  CONSENT 


I, _ ,  have  received  a  briefing  by  the  FAA 

representative  as  to  the  pmpose  of  the  FAA  study.  I  fully  understand  the  purpose  of  the  study 
and  have  been  provided  with  the  opportunity  to  ask  questions  of  the  FAA  representative.  The 
FAA  representative  informed  me  that  the  study  will  require  approximately  3  hours  of  training, 
followed  by  3  baggage  screening  trials,  and  30  minutes  to  complete  a  vision  test  and  1  hour  to 
complete  a  hidden  figures  test. 

I  understand  that  this  study  will  impose  very  little  stress.  The  only  stress  I  may  experience  in  this 
experiment  may  be  some  initial  frustration  as  I  learn  how  to  use  the  training  system.  As  part  of 
the  data  analysis,  my  data  will  be  combined  with  that  of  other  individuals  and  I  will  no  longer  be 
identifiable  as  a  participant.  I  have  been  informed  that  my  name  will  remain  CONFIDENTIAL. 

I  have  been  informed  that  I  have  the  right  to  withdraw  from  the  experiment,  and  that  the 
experiment  monitor  may  terminate  my  participation  in  the  interest  of  safety  and  the  experiment. 
I  also  certify  that  I  am  at  least  1 8  years  of  age. 

I  have  been  informed  that  if  additional  details  are  needed,  I  may  contact  any  of  the  test 
administrators  at  the  airport  during  the  study,  or  contact  James  L.  Fobes,  Ph.D.,  (609)  485-4944; 
or  Robert  L.  Malone,  (609)  645-0900,  upon  completion  of  the  study. 


Signed:  _ 

Date:  /  / 


Wimess:  _ 

Date:  _  /  / 
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SCREENER  PERSONAL  INFORMATION  FORM 
(SPEARS  OT&E  SCREENER  QUESTIONNAIRE) 
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FAA 


SCREENER  PROFICIENCY  EVALUATION  AND  REPORTING  SYSTEM 
OPERATIONAL  TEST  AND  EVALUATION 
SCREENER  QUESTIONNAIRE 

DATE:  _  SUBJECT  NUMBER: _ 

1.  What  is  your  sex? 

Male _  Female _ 

2.  How  long  have  you  been  a  baggage  screener? 

_ ^Years  _ ^Months 

3 .  How  long  have  you  been  using  X-ray  equipment  to  screen  baggage? 

_ Years  _ ^Months 

4.  -  What  is  the  highest  level  of  education  or  degree  that  you  have  completed. 

8th  Grade  or  Less  _ 

Some  High  School  _ 

High  School  Graduate  _ 

Some  College  or  University  _ 

College  or  University  Graduate  _ 

5.  Do  you  wear  glasses  while  using  the  X-ray  equipment? 
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Yes 


No 


M-3 


FAA 


SCREENER  PROFICIENCY  EVALUATION  AND  REPORTING  SYSTEM 
OPERATIONAL  TEST  AND  EVALUATION 
SCREENER  QUESTIONNAIRE 
(Continued) 

6.  Do  you  consider  English  to  be  your  primary  language? 

Yes _  No _ 

If  not,  in  which  language  are  you  most  proficient?  _ 

If  not,  are  you  proficient  with  English?  _ 
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PROTOCOL  IMPROVISED  EXPLOSIVE  DEVICE  (lED)  DETECTION  TEST 
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PROTOCOL 
lED  DETECTION  TEST 


READ  TO  SCREENERS: 

This  is  a  test  of  how  well  X-ray  machines  work  for  detecting  Improvised  Explosive  Devices 
(lEDs).  For  this  activity,  we  have  put  X-ray  images  of  passenger  bags  in  this  computer.  You 
will  view  the  X-ray  images,  one  bag  at  a  time,  and  inspect  each  bag  for  an  lED.  The  images  will 
be  displayed  on  the  video  monitor. 

INSTRUCTIONS  TO  SUBJECTS 

After  you  have  determined  that  your  subject  number  on  the  monitor  is  correct,  press  the 
“ENTER”  key  on  the  keyboard  to  start  the  practice  test.  Initiallyj  you  will  be  given  10  images  to 
practice  on  before  beginning  the  real  test.  Once  you  see  the  first  X-ray  image  appear  on  the 
monitor  screen,  your  task  will  be  twofold: 

First,  you  will  respond  to  the  question:  “Is  there  an  lED  in  this  X-ray  image?”  (The  question 
will  be  shown  at  the  bottom  center  of  the  monitor  frame.) 

To  indicate  “yes,”  press  the  key  labeled  “yes”  on  the  keyboard. 

To  indicate  “no,”  press  the  key  labeled  “no”  on  the  keyboard. 

Should  you  press  the  wrong  key,  an  error  message  will  appear  on  the  monitor  telling  you  that 
you  pressed  the  wrong  key. 

Because  we  want  to  know  how  well  the  machines  work,  it  is  important  that  you  answer  each 
question  to  the  best  of  your  ability.  It  is  just  as  important  for  you  to  say  “no”  when  you  do  not 
see  an  lED  as  it  is  to  say  “yes”  when  you  do  see  an  EED. 

You  will  have  10  seconds  to  make  your  response.  However,  you  should  answer  as  quickly  as 
possible.  Once  you  have  made  your  response,  you  will  not  allowed  to  go  back  and  change  it. 

If  you  have  not  answered  the  question  after  6  seconds,  an  audible  alarm  will  sound,  alerting  you 
that  you  have  only  4  seconds  left  to  answer  the  question. 

If  you  still  have  not  responded  after  10  seconds,  the  image  will  disappear  fi'om  the  monitor 
screen  and  will  be  replaced  by  a  prompt  which  advises  you  that,  “Time  has  expired,  please 
answer.” 
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After  you  have  responded  to  the  first  question,  a  second  question  will  be  displayed  on  the  video 
monitor: 


“How  sure  are  you?” 
yes  =  more  sure  no  =  less  sure 

Respond  by  pressing  the  “yes”  or  “no”  key  on  the  keyboard  or  keypad,  regardless  of  whether 
your  answer  to  the  first  question  was  “yes”  or  “no”.  In  either  case,  you  will  use  the  same  two 
keys  that  you  used  to  respond  to  the  first  question. 

You  will  have  5  seconds  to  make  this  second  response.  If  you  have  not  yet  answered  this  second 
question  after  5  seconds,  the  question  will  disappear  from  the  monitor  screen  and  the  prompt 
appear  again  telling  you:  “Time  has  expired,  please  answer.”  The  system  will  not  forward  to  the 
next  image  until  you  have  made  your  response. 

You  will  repeat  this  procedure  for  each  image  until  all  images  have  been  viewed.  A  message 
will  then  appear  on  the  monitor  indicating  that  the  test  has  ended.  The  test  should  take  a  little 
over  an  hour  to  complete. 

Do  you  have  any  questions? 

Thank  you  for  participating  in  this  test.  Press  the  “ENTER”  key  on  your  keyboard  to  start  the 
practice  test.  When  you  have  completed  the  practice  test,  press  the  “ENTER”  key  again  to  start 
the  real  test. 
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LOS  ANGELES  INTERNATIONAL  AIRPORT 
SITEMAPS 
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SPEARS  DETAILED  TEST  SCHEDULES 
(PRETEST,  POSTTEST  1,  AND  POSTTEST  2) 
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SPEARS  ON-LINE  OT&E  PRE-TEST 
TRAINING  GROUP 


DAY  4 


Legend: 


Vision  Testing 

Hidden  Figures/Hidden  Patterns 


BED  Testing 
TIP  Training 


SPEARS  ON-UNE  OT&E  PRETEST 
CONTROL  GROUP 


Documentation 


Hidden  Figures 


lED  Testing 
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SPEARS  ON-LINE  OT&E  POSTTEST 1 

training  group 


SPEARS  ON-UNE  OT&E  POSTTEST  1 
CONTROL  GROUP 


SPEARS  ON-LINE  OT&E  POSTTEST  2 


Note:  Training  Group  Day  1-2,  Control  Group  Day  3-5 
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REGAN  VISUAL  ACUITY  CHART  -  HIGH  CONTRAST  96  PERCENT 
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REGAN  HIGH  CONTRAST  ACUITY  CHARTS 


SCORE  SHEE 
SERIES  1 
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It  IS  important  to  urge  the  patient  to  guess  each  letter,  even  when  uncertain. 
Mark  each  error  by  crossing  out  each  letter  missed. 
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Appendix  R 
to  SPEARS  OT&E  TER 


SCREENER  INSTRUCTIONS 


SCREENER  OPERATING  INSTRUCTIONS  FOR  TnT™  TIP 


These  instructions  are  intended  to  help  you  operate  the  Linescan  System  E-Scan  X-ray  machine 
when  imdergoing  on-line  training  and  testing  during  this  study.  As  you  know,  the  X-ray  marhinff 
is  used  to  screen  hand-carried  parcels  to  identify  security  threats.  A  new  training  device  has  been 
developed  to  help  screeners  such  as  yourself  learn  to  use  the  E-Scan  X-ray  manhinp;  by  taking 
self-paced  training  lessons  and  on-line  training  and  testing  offered  by  the  Linescan  Training  and 
Testing  system.  (Throughout  the  rest  of  this  guide,  the  term  Linescan  TnT™  is  used  to  refer  to 
the  training  and  testing  system.) 

U.l  WHATTST.INESCANTnTTM? 

The  Linescan  TnT™  training  system  is  a  training  tool  developed  to  help  you  learn  how  to 
operate  the  E-Scan  X-ray  machine  and  how  to  identify  security  threats. 

The  training  system  itself  is  a  combination  of  equipment  and  software  (computer  programs). 

11.2  PURPOSE  OF  THESE  OPERATING  INSTRUCTIONS. 

The  purpose  of  these  operating  instructions  is  to  provide  you  with  all  the  information  you  need  to 
use  Linescan  TnT™  in  the  on-line  training  and  testing  mode.  With  it,  you  will  learn  how  to: 

a.  log  onto  the  system, 

b.  use  Linescan  TnT™  with  the  X-ray  machine  for  on-the-job  practice,  and 

c.  log  out  of  the  system. 

These  instructions  are  not  intended  to  teach  you  how  to  operate  the  E-Scan  X-ray  system  or  how 
to  identify  security  threats. 

U^XONTENTS  OF  THESE  INSTRUCTIONS. 

These  instructions  cover  the  topics  listed  below: 

a.  Starting  up  the  equipment. 

b.  Logging  onto  Linescan  TnT™. 

c.  The  Main  Menu  and  the  basics  of  using  Linescan  TnT™. 

d.  Instructions  for  using  Linescan  TnT™  when  it  is  connected  to  the  E-Scan  X-ray 
machine  for  on-line  training  and  testing  when  screening  for  parcels. 

e.  Logging  out  of  Linescan  TnT™. 
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UA  WHAT  IS  ON-LINE  TRAINING  AND  TESTING? 


During  on-line  training  and  testing,  you  practice  your  skills  in  threat  identification  while 
operating  the  actual  X-ray  machine.  You  also  learn  what  to  do  when  you  find  a  threat.  On-line 
training  and  testing  is  conducted  during  an  actual  work  shift. 

For  on-line  training  and  testing,  Linescan  TnT™  is  connected  to  the  E-Scan  X-ray  machine. 
Your  goal  is  to  identify  all  threats  that  appear  to  pass  through  the  X-ray  machine.  Linescan 
TnT™  then  asks  you  to  locate  the  specific  threat  you  see  in  the  X-ray  image  and  to  classify  it  as 
a  gun,  a  knife,  an  explosive  device,  or  a  miscellaneous  weapon.  Then  Linescan  TnT™  displays  a 
security  procedure  for  you  that  tells  you  how  to  handle  the  kind  of  threat  you  have  detected. 

In  on-line  testing,  your  performance  is  recorded  so  that  your  manager  can  check  your  progress  to 
see  where  your  strengths  and  weaknesses  are.  This  recording  feature  will  be  used  by  the  FAA 
representatives  to  see  how  well  the  Linescan  TnT™  trains  screeners  to  find  threats.  The 
recordings  are  confidential,  and  no  one  will  know  your  results. 

U.5  GETTING  STARTED. 

This  section  tells  you  how  to  log  onto  the  Linescan  TnT™  training  system  as  an  authorized  user. 
It  also  introduces  the  functions  available  to  you  as  an  operator  and  covers  the  basics  of  using  the 
Linescan  TnT™  screens.  Finally,  it  tells  you  how  to  log  out  at  the  end  of  your  work  session. 

U.5.1  Using  the  Trackball. 

The  most  important  part  of  learning  to  use  Linescan  TnT™  is  learning  to  use  the  trackball.  Take 
a  few  moments  now  to  become  familiar  with  using  it. 

Pointing 

The  trackball  is  a  pointing  device.  As  you  roll  the  trackball,  a  pointer  (usually  but  not  always  an 
arrow)  moves  on  the  screen  in  correspondence  with  the  trackball’s  movement.  You  use  the 
trackball  to  point  at  objects  on  the  screen. 

Sometimes  the  pointer  is  not  an  arrow.  Sometimes  it  takes  the  shape  of  a  pointing  hand  usually 
when  it  is  over  a  certain  type  of  object  on  the  screen.  Other  times,  the  pointer  takes  the  shape  of 
a  wrist  watch  with  hands  that  spin  clockwise.  This  “wrist  watch”  indicates  that  the  computer  is 
processing  your  request  and  you  should  wait  momentarily. 

Clicking 
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In  order  to  tell  Linescan  TnT™  what  you  want  it  to  do,  you  have  to  do  more  than  simply  point  at 
an  object.  To  select  the  object  (that  is,  to  tell  the  computer  you  want  to  use  that  particular 
object),  you  click  on  it. 

To  click  on  an  object  that  you  have  pointed  at,  press  the  pad  on  either  side  of  the  trackball.  You 
will  feel  a  slight  “clicking”  pressure,  almost  as  if  you  have  pressed  a  button,  and  you  will  hear  a 
“click.” 

You  use  the  actions  of  pointing  and  clicking  throughout  your  interaction  with  Linescan  TnT^. 
When  this  guide  instructs  you  to  select  or  to  “click”  or  “click  on”  an  object,  you  must  first  point 
at  the  object,  then  press  the  trackball  pad. 

LL5.2  Simulated  Control  Panel. 

You  will  often  see  on  a  Linescan  TnT™  screen  a  control  panel  that  looks  like  the  control  panel 
on  the  actual  TnT™  machine.  To  use  the  simulated  control  panel  on  the  screen  (figure  U-1),  you 
will  use  the  trackball  to  point  at  and  click  on  its  simulated  buttons.  In  particular,  you  will  click 
the  square  buttons  in  the  center  and  the  round  buttons  on  either  side.  You  will  not  click  on  the 
labels  next  to  the  roxmd  buttons. 


FIGURE  U-1.  SIMULATED  CONTROL  PANEL 


U.5,3  Logging  onto  Linescan  TnT™. 

Logging  on  is  the  process  of  identifying  yourself  to  Linescan  TnT™  as  an  authorized  user.  Only 
authorized  users  are  allowed  to  have  access  to  the  Linescan  TnT™. 

Before  you  log  on 

You  need  an  operator  identification  (“ID”)  number  and  a  password  in  order  to  log  in.  You 
should  have  been  given  your  operator  ID  and  a  password  by  your  manager. 
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When  you  log  on,  Linescan  TnT™  will  already  be  turned  on  and  the  login  screen  and  simulated 
control  panel  (figure  U-2)  will  be  displayed  on  the  left  monitor. 

NOTE:  If  the  login  screen  and  simulated  control  panel  are  not  displayed  on  the 
left  monitor,  see  your  manager. 

Logging  on 


a.  On  the  logon  screen  and  simulated  control  panel  (figure  U-2),  enter  your  operator 
ID  number.  To  do  so,  use  the  niunbered  buttons  on  the  simulated  control  panel. 

NOTE:  The  digits  1  through  9  are  on  the  square  buttons  in  the  center,  while  the 
digit  0  is  on  the  second  round  button  fi'om  the  top  to  the  right. 

1 .  Click  on  each  of  the  digits  in  the  ID  number.  (As  you  click  each  digit,  a 
beep  is  sounded.) 

2.  After  clicking  on  all  the  digits  in  the  ID,  click  the  ENTER  button. 


FIGURE  U-2.  LINESCAN  TNT™  LOGIN  SCREEN  AND  SIMULATED  CONTROL  PANEL 
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b.  You  can  now  use  your  password.  To  use  your  password,  click  on  each  of  the 
digits  in  the  password,  then  click  ENTER. 
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U.5.3  Logging  Out  of  Linescan  TnT™. 


Whenever  you  are  finished  using  Linescan  TnT™  or  whenever  you  need  to  leave  it  unattended, 
even  for  a  few  minutes,  you  should  log  out  (exit).  By  logging  out,  you  ensure  that  no 
unauthorized  person  can  access  Linescan  TnT™  information. 

Logging  out 


a.  Return  to  the  Main  Menu,  if  you  are  not  already  there. 

If  starting  from  some  other  Linescan  TnT™  screen,  click  EXIT  as  needed  until  you  return  to  the 
Main  Menu. 

b.  At  the  Main  Menu,  click  EXIT. 

After  a  few  moments,  the  Linescan  TnT™  login  screen  and  the  simulated  control  panel  appear 
on  the  left  monitor.  Linescan  TnT™  is  ready  for  another  authorized  user  to  log  in. 

NOTE:  If  you  are  also  responsible  for  turning  off  the  Linescan  TnT™  training 
station,  see  your  manager  for  instructions. 

ILOISING  X-RAY  MODE  tON-LINE  TRAINING  AND  TESTING^ 

This  section  tells  you  how  to  use  Linescan  TnT™  in  conjunction  with  the  E-Scan  X-ray  machine 
for  on-line  training  and  testing. 

U.6.  L  Overview  of  X-Rav  Mode. 

To  use  Linescan  TnT™  with  the  X-ray  machine,  you  need  to  know  about  X-ray  mode.  Threat 
Image  Projection  (TIP),  on-line  testing,  and  on-line  monitoring.  To  prepare  yourself  for  using  X- 
ray  mode  correctly,  read  the  following  overview  information  before  trying  to  use  the  X-ray 
mode. 

What  is  x-ray  mode? 

When  you  use  Linescan  TnT™  with  the  X-ray  machine  for  on-line  training  and  testing,  you  are 
using  X-ray  mode.  During  X-ray  mode,  you  use  the  actual  control  panel  on  the  X-ray  machine  to 
screen  parcels.  You  can  strip  the  inorganic  parts  of  an  X-ray  image,  change  it  to  reverse 
monochrome,  and  enlarge  the  image  to  get  a  closer  view. 

What  is  TIP? 

To  give  you  on-line  training  and/or  to  test  your  threat  identification  skills,  your  manager  can 
instruct  Linescan  TnT™  to  project  threat  images  into  actual  parcels  that  pass  through  the  X-ray 
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machine.  When  Linescan  TnT™  gets  ready  to  project  a  threat  image,  it  looks  for  a  parcel  that  is 
large  enough  to  hold  the  image  it  has  to  project.  For  example,  if  the  image  scheduled  to  be 
projected  is  a  long  item,  and  the  next  parcel  that  passes  through  the  X-ray  machine  is  a  small, 
square  package,  Linescan  Tn’n'^  does  not  project  the  image  into  that  parcel.  It  waits  until  it 
finds  a  parcel  whose  size  is  large  enough  to  hold  the  projected  image 

When  Linescan  TnT™  does  find  a  suitable  parcel,  it  projects  the  threat  image  into  the  X-ray 
image  of  the  actual  parcel.  As  far  as  you  can  tell,  the  projected  image  looks  as  real  in  the  X-ray 
as  the  parcel  itself. 

A  manager  typically  instructs  Linescan  TnT™  to  project  a  certain  number  of  images  during  a 
given  period  of  time,  such  as  a  work  shift.  For  example,  he  or  she  may  schedule  20  images  to  be 
projected  over  8  hours.  Approximately  every  half  hour  an  image  will  be  projected.  (However, 
remember  that  Linescan  TnT™  may  occasionally  not  project  a  threat  image  if  it  cannot  find  a 
parcel  large  enough  to  hold  it.) 

What  is  on-line  training  and  testing? 

On-line  training  and  testing  is  the  mode  in  which  an  X-ray  will  be  used.  In  on-line  training  and 
testing,  you  will  follow  a  procedure  with  several  steps  or  stages. 

When  you  think  you  have  detected  a  threat  (whether  a  projected  threat  or  a  real  one),  you  will 
press  the  TIP  button  on  the  actual  X-ray  machine.  (The  TIP  button  on  the  .simulated  control  panel 
of  Linescan  TnT™  is  active  as  well.) 

At  this  time,  you  are  informed  if  you  are  being  tested  or  not. 

If  the  threat  was  a  Projected  Threat  Image,  it  is  now  erased  firom  the  screen.  All  your  responses 
are  recorded  by  Linescan  TnT™  so  that  the  FAA  representatives  can  see  how  well  the  TIP 
fimction  works.  You  will  leam  more  about  this  on-line  testing  later  in  this  section. 

What.are  hits,  misses,  and  suspected  threats? 

A  hit  is  a  projected  threat  image  that  you  successfully  detected. 

A  miss  is  a  projected  threat  image  that  you  failed  to  detect.  A  miss  occurs  under  two 
circumstances: 


The  X-ray  into  which  the  image  was  projected  may  have  stayed  on  the  screen  for 
too  long  without  your  detecting  the  projected  threat.  The  amount  of  time  in  which 
you  have  to  respond  depends  on  how  Linescan  TnT™  has  been  set  up. 

The  X-ray  image  scrolled  (moved)  off  the  monitor  screen  before  you  pressed  the 
TIP  button. 
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A  suspected  threat  is  a  threat  that  you  perceived  to  be  in  the  parcel  but  that  was  not  projected  by 
Linescan  TnT™.  It  might  be  a  harmless  item,  or  it  might  be  a  real  threat.  When  this  happens 
during  on-line  monitoring,  Linescan  TnT™  automatically  saves  the  X-ray  so  that  you  and  your 
manager  can  review  it  later. 

U.6.2  Using  X-Rav  Mode. 

Starting  up  X-ray  mode. 

Before  you.  can  use  X-ray  mode,  Linescan  TnT™  must  be  connected  to  the  X-ray  machine.  Your 
manager  or  a  service  technician  will  have  already  connected  them. 

a.  On  the  Main  Menu,  click  X-ray  mode. 

NOTE:  If  you  see  the  following  message  and  Linescan  TnT™  is  not  connected  to 
the  X-ray  machine,  see  your  manager.  You  will  not  be  able  to  continue. 

PLEASE  MAKE  SURE  THE  X-RAY  MACHINE  IS  ON  AND 
ALL  CABLES  ARE  PROPERLY  CONNECTED 

b.  Now  operate  the  actual  X-ray  machine  as  you  have  been  trained  to  screen  parcels 
for  threats. 

Important !!:  The  threats  you  see  may  be  projected  threat  images 
or  they  may  be  actual  threats. 

c.  Continue  to  the  procedure  “Finding  a  threat  during  on-line  training  and  testing.” 
Finding  a  threat  during  on-line  training  and  testing. 

Recall  that  during  on-line  training  and  testing,  you  will  be  tested  with  threat  images  projected  by 
Linescan  TnT™. 

Important:  Read  this  entire  procedure  before  trying  it  to  learn  about  the  actions  you  will  be 
required  to  take.  If  you  were  to  find  an  actual  threat  in  a  parcel,  you  must  be  able  to  act  quickly. 

d.  If  you  think  you  see  a  threat  in  a  parcel  passing  through  the  X-ray  machine,  press 
the  TIP  button  on  the  actual  control  panel. 

If  the  threat  you  see... 

•  is  a  projected  threat  image,  the  following  message  appears  on  the  monitor.  After 
reading  it,  continue  with  Step  3. 
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YOU  HAVE  CORRECTLY  mENTIFIED  A  PROJECTED  THREAT! 

CLICK  ON  THE  THREAT  OB JECT. 

is  not  a  projected  threat  image,  the  following  message  appears  on  the  left  monitor: 


YOU  ARE  NOT  BEING  TESTED  AT  THIS  TIME. 

FOLLOW  THE  SECURITY  PROCEDURES  FOR  THE  SUSPECTED  THREAT! 
PRESS  DOWN  ON  THE  TRACKBALL  PAD  TO  CONTINUE. 


e.  When  you  are  ready  to  exit  X-ray  mode,  refer  to  “Exiting  X-ray  mode”  at  the  end 
of  this  section. 

U.6.3  Exiting  X-Rav  Mode. 


When  you  are  ready  to  exit  X-ray  mode  from  either  on-line  training  and  testing  or  on-line 
monitoring,  follow  this  procedure. 

a.  Press  the  TIP  button  on  the  TnT™. 

If  you  are  not  being  tested,  an  EXIT  button  appears  in  the  lower  left  comer  of  the  left  monitor  (as 
shown  in  the  example  in  figure  U-3). 

NOTE:  If  you  are  being  tested,  the  EXIT  button  does  not  appear.  You  must 
follow  the  standard  procedure  for  identifying  the  projected  threat.  Afterward,  you 
can  exit  X-ray  mode. 


YOU  Rtf  NOrSElWTCSTED  HTTIISTIHL 
FoiLinu  sEcaiTv  paodiiiiRES  for  ime  sKncTEonnEir  i 
PKII  onupc  DN  TVE IHUCH  PID  TD  CONTINUE 
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FIGURE  U-3.  EXIT  BUTTON  ON  AN  X-RAY  MODE  SCREEN 


b.  Click  the  EXIT  button  in  the  lower  left  comer.  (If  the  button  disappears  before 
you  have  a  chance  to  click  it,  just  repeat  6*001  Step  1 .) 


X-ray  mode  is  ended.  After  a  few  moments,  you  are  returned  to  the  Main  Menu. 


CARRY  OUT  THESE  STEPS  EACH  TIME  YOU  ARE  ASSIGNED 
TO  THE  X-RAY  MACHINE 


_ Ensure  the  log  on  screen  is  showing  on  the  left  hand  TnT™  monitor.  If  it  is  not 

on  the  left  hand  monitor,  see  your  supervisor. 

_ Enter  your  ID  number.  If  you  do  not  know  your  ID  number,  or  have  forgotten  it, 

see  your  supervisor.  Click  on  each  digit  of  your  ID  number  on  the  simulated 
control  panel  on  the  screen  using  the  trackball  and  the  clicking  pad.  After  you 
have  entered  in  all  digits,  press  ENTER. 

_ Using  the  trackball  and  the  clicking  pad,  press  X-ray  Mode  on  the  Main  Menu 

-  Operate  the  actual  X-ray  machine  as  you  have  been  trained  to  screen  parcels  for 

threats. 


CARRY  OUT  THESE  STEPS  EACH  TIME  YOU  LEAVE  THE  X-RAY  MACHINE 

- Press  the  TIP  button.  If  you  are  not  being  tested,  an  EXIT  button  will  appear  in 

the  lower  left  comer  of  the  monitor.  If  you  are  being  tested,  the  EXIT  button  will 
not  appear.  See  your  manager. 

- Using  the  trackball  and  the  clicking  pad,  press  the  EXIT  button  on  the  monitor 

until  you  reach  the  main  menu. 

_  Using  the  trackball  and  clicking  pad,  press  EXIT  on  the  main  menu. 
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THE  LINESCAN®  TESTING  AND  TRAINING  (TnT)  OVERVIEW 


The  Linescan®  Tn'F”  system  is  a  computer-based  training  tool  developed  by  EG&G 
Astrophysics  to  help  baggage  screeners  operate  Linescan®  X-ray  equipment  with  E-Scan  and 
identify  security  threats.  The  training  system  provides  four  distinct  types  of  training  and  testing: 
training  lessons,  real-time  testing,  real-time  monitoring,  and  ongoing  training  and  testing  The 
Operational  Test  and  Evaluation  (OT&E)  presented  baggage  screeners  with  customized  training 
lessons  on  detecting  Improvised  Explosive  Devices  (lEDs). 

The  Tn'F”  offers  on-line  and  off-line  operations.  On-line  operation  provides  real-time  training 
and  testing  where  operators  can  practice  their  skills  in  threat  identification  while  operating  an 
actual  X-ray  machine.  Off-line  training  allows  operators  to  leam  and  practice  ttoeat 
identification  skills  by  taking  training  lessons  without  operating  the  X-ray  machine. 

Baggage  screeners  received  off-line  training  to  identify  threats  using  black/white  and  enhanced 
X-ray  equipment. 

The  TnT™  system  is  used  to  train  baggage  screeners  to  improve  performance  in  detecting 
threatening  objects.  Screeners  were  required  to  identify  threats  using  both  black/white  and 
enhanced  X-ray  equipment. 

Linescan^-TnF*^  System  Components  and  Operation. 

The  TnT™  system  consists  of  a  combination  of  equipment  and  software.  The  equipment  consists 
of  the  following: 

a.  A  color  monitor  and  a  black-and-white  monitor  (mounted  side-by-side,  just  as  the 
monitors  are  mounted  on  the  actual  E-Scan  X-ray  machine). 

b.  A  control  panel  (mounted  below  the  monitors)  that  replicates  the  control  panel  on 
the  actual  E-Scan  X-ray  machine 

c.  A  trackball  (located  to  the  right  of  the  control  panel)  with  which  operators  interact 
with  the  training  programs. 

d.  A  computer  (located  inside  the  equipment  housing)  that  runs  the  training 
programs. 

e.  A  variety  of  cables  and  connectors  used  to  connect  the  training  system  to  the  X- 
ray  and  to  recording  equipment  such  as  camcorders  and  video  cassette  recorders. 

During  the  OT&E,  participating  baggage  screeners  received  seven  pre-determined  lessons  from 
the  Tn’F“  Lessons  menu.  All  lessons  were  previously  completed  at  the  Federal  Aviation 
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Administration  (FAA)  Technical  Center  prior  to  conducting  the  OT&E.  The  FAA  selected  the 
training  lessons  for  the  OT&E.  A  copy  of  the  TnP^“  operating  instructions  was  located  on  the 
Tn'F”,  and  an  FAA  representative  monitored  all  screeners  operating  the  TnT™. 

All  Linescan®  TnT™  lessons  were  conducted  in  an  office  behind  the  United  Airlines  group  check 
check-in  counter. 

The  following  instructions  were  provided  to  baggage  screeners  during  the  TnT™  training  An 
FAA  representative  was  present  during  the  training  and  provided  baggage  screeners  with 
assistance  when  necessary. 

LINESCAN^  TRAINING  AND  TESTING  (Tn-p  INSTRUCTIONS. 

The  Linescan®  TnT  training  system  is  used  to  train  baggage  screeners  to  improve  performance  in 
detecting  threat  objects.  By  taking  the  following  training  lessons,  one  will  learn  to  identify 
threats  using  both  black/white  and  enhanced  x-ray  equipment.  Review  the  seven  training  lessons 
identified  in  the  Training  Lessons  Procedures  list.  Please  do  not  take  lessons  that  are  not 
included  on  the  list. 

If  there  are  questions  at  any  time,  ask  the  Federal  Aviation  Administration  (FAA) 
representative  for  assistance. 

Getting  Started. 

If  there  are  questions  about  beginning  the  training  lessons,  turning  the  equipment  on,  or  using 
available  functions,  ask  the  FAA  representative  for  assistance. 

Use  the  following  procedures  to  begin  the  training; 

Logon  Procedures. 

1.  .  Using  trackball,  position  the  arrow  on  the  visual  display  number  pad. 

2.  Using  the  visual  display  number  pad,  enter  the  operator  ID:  123 

3.  Press  the  ENTER  button  to  enter  operator  ID. 

4.  Using  the  visual  display  number  pad,  enter  password:  123 

5.  Press  the  ENTER  button  to  enter  password. 

6.  Press  the  CLEAR  button  if  a  mistake  is  made  and  to  restart  the  logon  procedures. 

7.  Press  the  TIP  button  on  the  TnT  control  panel. 
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Reviewing  a  T  .fi5;gfiTi 


After  successfully  logging  onto  the  training  system,  complete  the  following  steps: 

1 .  Select  the  Lessons  button  from  the  opening  screen. 

The  Lessons  menu  appears. 

2.  On  the  Lessons  menu,  click  Review. 

The  Review  menu  appears. 

J .  To  review  a  prior  lesson,  click  A  Prior  Lesson. 

A  checklist  of  the  lessons  previously  completed  is  displayed.  It  shows  up 
to  10  lessons  at  a  time.  To  bring  other  lessons  into  view,  use  the  up  and 
down  arrows  to  the  right  of  the  lessons. 

To  select  the  lesson  to  review,  click  on  it. 

Table  W-1  shows  a  list  of  the  lessons  to  review.  Review  the  lessons  in  the 
order  presented  in  table  W-1. 

TABLE  W-1.  LIST  OF  LESSONS  TO  REVIEW 


Unit  2 

Lesson  1 

Unit  4 

Lesson  1 

Unit  4 

Lesson  3 

Unit  5 

Lesson  4 

Unit  6 

!  Lesson  1 

Unit  6 

Lesson  3 

Unit? 

Lesson  1 

When  the  lesson  is  presented,  complete  it  by  closely  reading  the 
information  presented  on  the  screens. 

5.  Begm  the  appropriate  lesson  displayed  on  the  right  visual  display  (Lesson  Screen). 

6.  When  finished  reviewing  a  lesson,  immgdiatgly  review  the  corresponding  test. 
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Appendix  U 
to  SPEARS  OT&E  TER 


TEST  BAG  CONFIGURATIONS 
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Requests  for  descriptions  of  test  bag  configurations  should  be  submitted  in  writing 
to  the  FAA  program  manager  supporting  this  effort. 
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Appendix  V 
to  SPEARS  OT&E  TER 


CHECKED-CARRY-ON  BAGGAGE  -  THREAT  ARTICLE  DEFINITION  SHEETS 
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Requests  for  descriptions  of  threat  article  definitions  should  be  submitted  in 
writing  to  the  FAA  program  manager  supporting  this  effort. 


V-2 


Appendix  W 
to  SPEARS  OT&E  TER 


THREAT-IMAGE  PROJECTION  INSERTION  PROTOCOL 
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TIP  1  INSERTION  PROTOCOL 
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TIP  2  INSERTION  PROTOCOL 
WEEKS  1-3 


TIP  2  INSERTION 


r 
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TIP  2  INSERTION  PROTOCOL 
WEEKS  7-8 


Appendix  X 
to  SPEARS  OT&E  TER 


NORTHWEST  AIRLINES  DAILY  DEPARTURES  -  LOS  ANGELES 
INTERNATIONAL  AIRPORT 
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NORTHWEST  AIRUNES  DAILY  DEPARTURES 
LOS  ANGELES  INTERNATIONAL  AIRPORT 


Legend: 

AMS 

Amsterdam 

BOS 

Boston 

DTW 

Detroit 

FAT 

Fresno 

HNL 

Honolulu 

KIX 

Osaka 

LAS 

Las  Vegas 

LGA 

LaGuardia 

LGW 

Gatwick 

MEM 

Memphis 

MIA 

Miami 

MKE 

Milwaukee 

MNL 

Manila 

MRY 

Monterey 

MSP 

Minneapolis 

NGO 

Nagaro 

NRT 

Narita 

ONT 

Ontario 

PEK 

Beejing 

PHX 

Pheonix 

PSP 

Palm  Springs 

RSW 

Fort  Meyers 

SAN 

San  Diego 

SBA 

Santa  Barbara 

SBP 

San  Luis  Obispo 

SEA 

Seattle 

SEL 

Seoul 

SFO 

San  Francisco 

SHA 

Shangai 

YYZ 

Toronto 

Note:  Northwest  Airlines 
accounts  for  approximately 
75%  of  enplanements 
using  this  checkpoint 
Other  airlines  using  this 
terminal  checkpoint  Include 
Virgin  Air,  Asiana,  Air 
Canada,  Hawaiian  Air, 
Vast,  and  Air  New  Zealand. 
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